QSFP28 Troubleshooting: Fix 100G Link Down & Errors

Modules get returned that test good at the factory. The RMA takes two weeks, costs shipping both ways, and the link is still down when the replacement arrives - because the fault was never in the optic. It was a contaminated ferrule, an FEC mismatch, or a port that was never put into breakout mode.

The reason this keeps happening is not a lack of knowledge. It is sequence. At 100G there are enough plausible causes that starting in the wrong place burns days, and every component you swap without recording the result destroys a little more of the evidence you need.

This guide runs five layers in a fixed order, from the cheapest thing to check to the most expensive. Work them in order and the fault usually names itself.

The Fault Order: Why the Module Is Usually Innocent

You will find plenty of articles claiming a specific percentage of 100G failures are physical-layer problems. Treat those numbers with suspicion - nobody publishes the dataset, and the figures vary wildly between sources that all sound equally confident. What is defensible is the ordering, and the ordering is what actually saves you time:

1. Physical - seating, contamination, connector and cable damage. Cheapest to check, fastest to fix, and the thing most often skipped because it feels too obvious.
2. Identity - can the host read the module at all, and does it support it? A software problem that presents as a hardware problem.
3. Configuration - speed, FEC, breakout, both ends. The classic "good light, no link".
4. Signal quality - DDM, optical budget, error counters, temperature. Where marginal links live before they fail.
5. Isolation - controlled substitution, when the first four have not answered it.

Each layer assumes the one above it is clean. Do not analyze BER on a link whose connector you have not inspected; you will be measuring dirt.

Practical takeaway: do not replace the module until the cable, port, configuration and remote end have been ruled out. A module that "fails" in one port and works in the next was never faulty.

Start Here: Identify the Link Type

This step comes before any measurement, and it is the one most workflows skip. QSFP28 is a form factor, not an optical interface. The valid tests differ by module - running an MPO polarity check on a duplex LC link is not a wasted hour, it is a wrong answer.

SR4 and parallel multimode

100GBASE-SR4 uses parallel multimode fiber over MPO-12: four transmit lanes, four receive lanes, one 100G link. Usual suspects, most common first: contamination on one or more MPO fibers, incorrect polarity, damaged guide pins or ferrules, reach exceeded, uneven RX between lanes, a partially mated breakout path.

One abnormal lane destabilizes the entire interface while the other three read perfectly normal. If the assembly mixes connector types or polarity methods, confirm it against an MTP and MPO selection guide before assuming the module is at fault - and check the run against real OM3 and OM4 distance limits, which differ enough to matter across a long row.

LR4, CWDM4 and duplex single-mode

Multiple wavelengths combined inside the module, carried over one TX and one RX fiber on duplex LC. Usual suspects: TX and RX reversed, excess connector or splice loss, multimode patch cords in a single-mode path or an OS1 cable on a route designed for OS2, RX overload on a very short link, a wavelength-channel fault inside the module, incompatible FEC or host electrical interface.

DR1, FR1 and LR1 single-lambda

One high-speed PAM4 optical lane instead of four 25G lanes. Same cage, different electrical and FEC behavior. Confirm the port, line card, operating system and FEC mode support the exact module - the mechanical fit tells you nothing about the electrical fit. These types also report diagnostics differently: Arista notes that 100GBASE-FR, LR and DR do not publish thresholds for enhanced DOM parameters, so an empty threshold column on a single-lambda optic is expected behavior, not a fault.

DAC, AOC and 4×25G breakout

DAC is copper with no serviceable optical interface. AOC has permanently attached fiber. A QSFP28-to-4×SFP28 assembly splits one port into four 25G interfaces. Focus on: part-number compatibility, supported length, parent-port breakout mode, lane mapping, FEC on all four child interfaces, and remote 25G port configuration. Where the split happens in the fiber plant rather than the cable, the same MPO breakout cable selection rules govern lane mapping.

Juniper's Hardware Compatibility Tool distinguishes single-lane, dual-lane and four-lane 100G interfaces along with their fiber, connector and host-interface requirements - which is precisely why identifying the optic has to come before interpreting any FEC or DDM reading.

QSFP28 Symptom

Causes ordered by how often they turn up in the field, not alphabetically. Start at the top of each cell.

QSFP28 Troubleshooting Decision Tree

• Is the module detected?
  • No → reseat, check power class and software support, test the module in a known-good port and a known-good module in the suspect port. See Layer 2.
  • Yes → continue.
• Is RX power present and inside the datasheet window at both ends?
  • Low → inspect and clean, then compare against remote TX and the path loss calculation.
  • High → long-reach optic on a short link; compare against the overload limit.
  • Normal → continue.
• Do speed, FEC and breakout mode match at both ends?
  • No → fix the mismatch first. This is the leading cause of "good light, no link". See Layer 3.
  • Yes → continue.
• Are errors increasing under real traffic?
  • Uncorrectable FEC codewords present → physical-layer fault; inspect and isolate now.
  • Corrected FEC rising against baseline → margin problem; clean, then compare lanes and both ends.
  • No → healthy link. Record the baseline and stop.
• Which component does the fault follow?
  • The cable → the assembly, its connectors, its polarity.
  • The module → it becomes the leading RMA candidate.
  • The port → port, ASIC lane, line card, port profile.
  • Nothing follows; it stays with the link → the path and the remote end.

Layer 1: Physical Verification

The cheapest layer, and the one that gets skipped because engineers assume someone already checked. Nobody checked.

Visual inspection

• Seating. Push until the latch clicks. Partial insertion produces intermittent lane errors that look exactly like a dying module.
• Electrical contacts. Corrosion, debris or bent pins on the module edge connector. One damaged contact can take out one or more 25G lanes.
• Connector condition. Cracked ferrules, pulled boots, damaged MPO guide pins, kinked cable.
• Dust caps. A module or patch cord stored uncapped should be inspected before it goes anywhere near a port.

Inspect, clean, inspect

Never clean what you have not inspected - you have no idea whether cleaning helped, and you may have made it worse.

1. Disable or safely disconnect the path per site procedure.
2. Inspect the end face with a fiber microscope or inspection probe.
3. Clean with a tool designed for that connector type - an MPO cleaner for MPO, an appropriate method for duplex LC. Follow the fluid and connector manufacturer's instructions rather than applying one solvent rule everywhere.
4. Inspect again.
5. Clean the mating adapter or transceiver receptacle where required.
6. Reconnect and retest.

Pass and fail here are not a matter of opinion. IEC 61300-3-35 defines the inspection zones, the quantitative criteria for scratches, debris and defects, and the minimum microscope requirements. If your probe reports a grade, that standard is what it is grading against. On duplex links the tolerance for a marginal end face is smaller than most people assume - see how LC connector loss and return loss degrade once the ferrule is anything less than clean.

Never look directly into an active optical port or fiber. Follow the laser safety instructions in your platform and module documentation.

Cable and environment

Bend radius. Use the cable manufacturer's specified installation and long-term radii. There is no single universal figure that applies to every assembly, and tight cable management is a common source of micro-bending loss that shows up as unexplained attenuation - the practical limits are worth checking against fiber optic cable installation practice rather than guessed at.

Strain relief. A heavy MPO trunk hanging off a module causes intermittent contact that no amount of CLI will reveal.

Airflow. Modules in the upper rows of a dense chassis ingest pre-heated air and run measurably hotter than lower ports. Missing blanking panels let air bypass the chassis entirely.

Layer 2: Module Identity and Compatibility

Why is my QSFP28 not detected?

"Not detected" is four different problems wearing one label: absent, invalid, unsupported, unknown. They are not interchangeable, and the switch is not always telling you the truth about which one it is.

QSFP28 modules expose identification and diagnostics through a standardized management memory map. For QSFP-class optics that map is defined by SFF-8636, maintained by the SFF Technical Work Group; the newer CMIS interface covers later form factors. Both live in the published SFF specifications. In SFF-8636's upper page 00h, the fields the host reads at insertion include:

If nothing appears at all:

1. Reseat the module.
2. Test the same module in a compatible known-good port.
3. Test a qualified known-good module in the suspect port.
4. Check whether the port supports that module type and power class.
5. Review the switch software requirements and release notes for the platform.
6. Inspect the electrical contact area.

That fifth step deserves more respect than it usually gets. Platform vendors publish known issues affecting optics inventory and detection on specific software releases and line cards. Reading the release notes for your exact version before shipping hardware back is fifteen minutes against a two-week RMA cycle.

Compatibility and vendor coding

If the module appears but is flagged unsupported, check the compatibility matrix before reaching for hidden commands. Cisco publishes the Transceiver Module Group compatibility matrix for optic-to-device support; Juniper publishes the Hardware Compatibility Tool, which also tells you whether a given transceiver supports optical monitoring at all - useful before you spend an hour hunting for DDM fields that were never going to appear.

Platforms differ in how strictly they validate the vendor fields above, and behavior changes between software releases. Juniper states plainly that JTAC does not support third-party optical modules and cables it has not qualified, and that damage to host equipment caused by third-party optics - particularly high-power modules - is the user's responsibility. Whatever your sourcing policy, the practical point for troubleshooting is narrow: a module rejected on vendor coding is not an optically faulty module, and swapping it for another unit of the same part number will reproduce the same result exactly.

Example diagnostic commands

Commands differ by platform and software release. Verify syntax against the documentation for your exact device and version.

Cisco

• show interfaces transceiver detail
• show idprom interface <interface-id>
• show interface status

Juniper

• show interfaces diagnostics optics <interface-name>
• show interfaces <interface-name> extensive
• show chassis hardware

Arista

• show interfaces transceiver dom
• show interfaces transceiver dom thresholds
• show interfaces status

NVIDIA Cumulus Linux

• nv show platform transceiver <interface-id>
• nv show interface <interface-id> counters errors
• nv show interface <interface-id> link phy-detail

One detail worth knowing before you read any of that output: NVIDIA's Cumulus Linux transceiver monitoring guide prints each measured value directly alongside the module's own high and low alarm and warning thresholds. Those are the numbers to compare against - not a figure you remember from a different optic.

Layer 3: Configuration

Speed, FEC and breakout

The module is detected, the fiber is clean, the light is good, the link is down. Check FEC.

There is no single correct FEC setting for every QSFP28 link, and any guide that gives you one is wrong. The terms, quickly:

• FEC - forward error correction adds redundancy so the receiver repairs a bounded number of bit errors without retransmission.
• RS-FEC - the Reed–Solomon variant required by several 100G and 25G physical layers. KR4 and CAUI-4 name different host electrical interfaces between switch ASIC and module cage.
• PAM4 - two bits per symbol, which is what enables single-lambda 100G and also means less signal margin, so its FEC requirements differ from four-lane NRZ optics.

Which combination applies depends on the optic, the host ASIC, the Ethernet standard and the platform implementation. The definitions are in the IEEE 802.3 Ethernet standard; what your specific box does with them is in its configuration guide. Check both ends against both.

Verify at both ends: port speed, FEC on or off, FEC type, autonegotiation state, breakout mode, child-interface speed, module operating mode.

A mismatch produces link down with valid optical power, high error counts, a link that comes up at one end only, repeated flaps, or partial breakout operation.

Troubleshooting a 4×25G breakout

When only some child lanes come up:

1. Confirm the parent QSFP28 port is configured for 4×25G. A parent port that was never split explains all four lanes down, immediately.
2. Verify all four remote ports are enabled and set to 25G.
3. Compare FEC on each child interface - they can drift independently.
4. Confirm the breakout cable part number and its documented lane mapping.
5. Move the failed leg to a known-good SFP28 port.
6. Determine whether the fault follows the cable leg, the remote module, or the switch lane.

On polarity: verify the actual method against the cable's own documentation and your switch pinout. Assemblies vary, and a confident guess about which method "breakout cables normally use" is how a three-of-four-lanes problem survives an afternoon.

Do not swap all four remote modules at once. One variable per test, or the result carries no information.

Layer 4: Signal Quality, DDM and Thermal

Reading DDM

DDM (or DOM) reports module temperature, supply voltage, TX and RX optical power, laser bias current, and alarm and warning flags. Every one of those values is meaningless in isolation. Compare against four references:

1. The current value
2. The module's own warning and alarm thresholds, as the module itself reports them
3. The datasheet operating limits
4. A historical baseline from the same link or a comparable working port

Reference two is the one people skip, and it is why universal QSFP28 temperature or power ranges do not exist. A number that alarms on one optic is nominal on another.

The bias trend is your best early warning. A laser drawing steadily more current to hold the same output power is aging. That is a scheduled swap, not a 3 a.m. page - but only if someone recorded the baseline months ago.

What causes a QSFP28 TX fault?

TX fault means the transmitter has shut down. In order of likelihood: over-temperature triggering the module's own laser protection, laser bias current out of range, host control or electrical interface problems, and finally an actual transmitter failure. Check temperature first, every time - the specific shutdown threshold is a property of the module, so read it from the module rather than from a rule of thumb.

The thermal cascade: one fault wearing three masks

This is the pattern that wastes the most time, because it does not look like one problem:

1. Ambient rises - summer, a failed HVAC unit, a row rebalance that nobody modeled.
2. Module temperature crosses its own threshold.
3. Laser protection triggers → TX fault locally.
4. The remote end loses light → RX LOS there.
5. Thermal protection disrupts management-interface reads → the switch may report the module as unsupported or absent.

Three tickets, three engineers, three theories. One thermal event. If you are looking at a TX fault, an LOS at the far end and a detection problem in the same window, stop treating them as independent and go read the inlet temperature.

Low RX power: a worked optical power budget

Most guides print the formula and move on. The formula is only useful if you actually run it, and running it requires getting the units right:

Expected RX power (dBm) ≈ Remote TX power (dBm) − total path loss (dB)

Power is dBm. Loss is dB. Subtracting dB from dBm yields dBm. Take a 2 km OS2 duplex link, LR4 at each end:

• Remote TX power, from the far-end DDM readout: −1.5 dBm
• Fiber attenuation: 2 km × 0.35 dB/km = 0.7 dB
• Four mated connector pairs (two panels, two equipment ends) at 0.5 dB each = 2.0 dB
• Two fusion splices at 0.1 dB each = 0.2 dB
• Total path loss: 2.9 dB

Expected local RX ≈ −1.5 − 2.9 = −4.4 dBm.

Now compare against two datasheet limits, not one:

• Minimum receiver sensitivity - often around −8.6 dBm for a 100GBASE-LR4 module. −4.4 dBm sits roughly 4 dB above it. Comfortable.
• Receiver overload - often around +4.5 dBm. Nowhere near it, so no attenuator.

Suppose the port actually reads −9.8 dBm. The gap between calculated and measured is 5.4 dB, and that gap is the fault. Five decibels does not materialize on a 2 km run: it is a contaminated end face, an unaccounted patch panel, a bend, or a fiber that is not the one on the drawing. Chase the discrepancy, not the module.

If your loss figures come from a design document rather than a test set, validate the assumptions first - the delta between calculated and field-measured attenuation is exactly where insertion loss in real fiber networks tends to surprise people.

Per-lane patterns

• One low RX lane → contaminated MPO fiber or a damaged lane.
• Several similar low lanes → total path loss.
• One low TX lane that follows the module → transmitter fault.
• Lane imbalance that follows the cable → polarity, fiber or connector damage.

WDM modules may expose aggregate or per-channel data depending on module and platform. A missing field is not automatically a fault.

Rising FEC and BER counters

A link can stay up while quietly running out of margin, and the counters show it first. BER - bit error ratio - is the fraction of received bits that arrive wrong. Pre-FEC BER is measured before correction; effective or post-FEC BER is what survives it.

Monitor corrected FEC errors, uncorrectable codewords, effective BER, CRC errors, symbol errors, lane errors, loss-of-signal events, link transitions.

Corrected FEC errors do not mean the link failed. Correcting errors is what FEC is for. The question is whether the corrected error rate is climbing against its own baseline. Uncorrectable errors are categorically different: the correction system could not recover that data, and the link is one noise burst from flapping.

1. Save the current counters.
2. Clear software counters if appropriate.
3. Apply representative traffic - idle links hide marginal errors.
4. Observe the rate over a defined period.
5. Compare all lanes and both ends.
6. Repeat after cleaning or changing one component.

Per-lane tooling earns its keep here. Cumulus Linux exposes per-lane physical-layer error counters and SNR through nv show interface <interface> link phy-detail; Arista reports pre-FEC BER, uncorrected BER and SNR per channel in its DOM output. "The link has errors" is a week of work. "Lane 3 has errors" is an afternoon.

Layer 5: Controlled Isolation

When the first four layers have not answered it, substitute deliberately - one variable at a time.

The swap sequence

1. Suspect module → known-good port. Works there → the original port or cable. Still fails → the module leads the suspect list.
2. Known-good module → suspect port. Works → original module. Fails → port or cable.
3. Known-good cable. Link comes up → the cable. Still down → port or module.
4. Remote end. If both local tests pass, repeat steps 1–2 at the far end. A local RX problem is frequently caused by the remote TX path, and no amount of testing at your end will show it.

Two rules make this work. Your "known-good" reference must be qualified on this platform - an unqualified spare proves nothing. And one component moves per test; simultaneous swaps destroy diagnostic clarity and you will end up repeating the whole sequence.

Loopback testing

A loopback returns TX to RX internally, separating host-side from fiber-side in about thirty seconds. Insert it, enable the port: the port should come up with no fiber attached, DOM will show high RX power (expected - it is a short loop), and BER should be near zero.

Port will not come up with a loopback → host side: ASIC, SerDes, electrical interface, firmware, configuration. (SerDes is the serializer/deserializer that converts between the switch's internal parallel data and the module's serial lanes; a port-bound fault is often electrical rather than optical.) Works with loopback but not with a real module and fiber → optical, cabling, remote-side or interoperability. It narrows the search rather than ending it: a loopback validates neither the production fiber route nor interoperability with the remote device.

When Should You Replace or RMA a QSFP28 Module?

A module is a reasonable RMA candidate when the fault consistently follows it, and only after you have confirmed: the port supports it; the software version is compatible; optic and fiber types match; connectors have been inspected and cleaned; speed and FEC are correct; a known-good cable does not fix it; the module fails in more than one compatible known-good port; a known-good module works in the original link; and DDM or alarm behavior stays abnormal under valid operating conditions.

Before contacting the supplier, collect:

• Switch brand, model, line card, software version
• Interface name and configuration
• Module part number, serial number, coding
• Local and remote DDM output
• FEC and interface counters
• Fiber type, connector, distance, patching path
• Fault timestamps and recent changes
• Results of the cable, module, port and remote-end swaps

This is what lets a supplier reproduce the configuration and separate a module failure from an interoperability problem. Without it, the conversation restarts at zero and the module comes back marked "tested good" - which is true, and useless.

Common QSFP28 Troubleshooting Mistakes

• Replacing several components at once, so no result is attributable
• Applying universal temperature or optical-power thresholds instead of the module's own
• Checking only one end of the link
• Ignoring FEC because the optical power looks fine
• Assuming every 100G QSFP28 uses four optical lanes
• Running MPO polarity tests on a duplex LC link
• Mixing multimode and single-mode patching
• Using an unqualified spare as the known-good reference
• Clearing counters before recording them
• Treating corrected FEC errors as equivalent to uncorrectable ones
• Returning a module that was never tested against a different cable and port

How to Prevent Repeat 100G Link Failures

1. Validate the module against the switch and software compatibility matrix before it ships.
2. Confirm optic type, connector, fiber and reach together - not separately.
3. Document the required FEC and breakout settings alongside the port design.
4. Inspect and clean every optical connection at install time. Every one.
5. Record baseline DDM at both ends while the link is healthy. You cannot detect a trend you never started measuring.
6. Record clean interface and FEC counters as part of handover.
7. Label parent ports, child lanes and cable polarity - especially on MPO breakout assemblies, where a mislabeled leg costs far more time than the cable costs money.
8. Qualify spare modules on the actual target platforms, not on a lab switch that resembles them.
9. Alert on trends, not instants: temperature approaching the module's own threshold, power drifting from baseline, any uncorrectable FEC.
10. Keep firmware and configuration changes traceable.

QSFP28 Troubleshooting FAQ

Why is my QSFP28 not detected?

Check seating first, then compatibility. The module may be incompletely seated, unsupported by the port or its power class, incompatible with the current software release, coded for a different platform, unable to communicate through its management interface, or faulty - in roughly that order. Reseat, read the inventory, check the compatibility matrix, review the release notes for your software version, and test it in a compatible known-good port before shipping anything back.

What causes a QSFP28 TX fault?

The transmitter has shut down, most often because the module hit its own over-temperature protection. Less commonly: laser bias out of range, host control problems, or genuine transmitter failure. Read the module temperature and its advertised threshold before anything else. If a TX fault, a far-end LOS and a detection problem all appear together, you are almost certainly looking at one thermal event, not three faults.

Why is the link down when RX power is normal?

Optical power only confirms the receiver detects light. The link can still stay down because of FEC, speed, breakout mode, negotiation, lane alignment, module interoperability or remote-port configuration. This is the most common "good light, no link" scenario, and the fix is nearly always a configuration comparison between the two ends.

What causes low RX power on a QSFP28 link?

In rough order: a dirty connector, incorrect patching, excess connector or splice loss, excessive distance, weak remote TX, damaged fiber, and finally the module receiver. Compare remote TX against local RX and against a calculated path loss before replacing anything. The size of the gap between calculated and measured tells you what kind of fault you are looking for.

Does every QSFP28 module require FEC?

No. FEC requirements depend on the module's host electrical interface, its optical standard and the switch implementation. Four-lane NRZ optics and single-lambda PAM4 optics can behave differently in the same switch. Check the optic datasheet and the platform configuration guide for both ends, and be sceptical of any blanket rule.

Can DDM prove a module is faulty?

Rarely on its own. DDM reveals abnormal temperature, voltage, TX or RX power, bias current or alarm flags, but a single reading proves very little. Compare against the module's own thresholds, its history, the remote end, and a known-good module.

Why does only one 25G breakout lane fail?

The fault sits in one cable leg, one SFP28 module, one remote port, one host lane, the lane mapping, or the FEC configuration of that child interface. Confirm the parent port is actually split, then swap one leg at a time and watch where the failure follows.

When should I replace the QSFP28 module?

When the fault follows the module across compatible known-good ports while a qualified known-good module works with the same cable, configuration and remote device. Anything short of that is a guess with shipping costs attached.

Conclusion

QSFP28 troubleshooting is mostly discipline. The modules are mature, the standards are settled, and the failure patterns are well understood - what goes wrong is the order of investigation and the destruction of evidence along the way.

Identify the link type first, because it decides which tests are even valid. Then work down: physical, identity, configuration, signal, isolation. Compare both ends. Change one variable at a time. Record the numbers before you clear them.

And before ordering a replacement, have the switch model, software version, optic type, link distance, fiber type, FEC mode, DDM readings and error counters ready. Compatibility questions get answered in minutes with the full link picture, and in weeks without it.