Problems of frame loss, jitter and latency using SWI rev D

BotBlox Hardware
1

This topic was migrated from the BotBlox ticketing system after being summarised and anonymized.

Problem:

Customer observed heavy frame loss, jitter, latency, and Ethernet link instability when using SwitchBlox Industrial Rev D, while the same setup worked correctly with Rev B hardware. The issue was especially apparent when tested with an EXFO Ethernet tester and did not occur on previous revisions.

Answer:

Issue was traced to the clock generation circuit introduced in Rev D, specifically the switch from a quartz crystal oscillator to a MEMS oscillator. Extensive testing showed the problem disappeared when an external 25 MHz reference clock was used. A higher-performance silicon oscillator (LMK6CE02500CDLFR) completely resolved the issue during testing.

However, subsequent BERT testing indicated the oscillator change alone may not fully explain the problem. Due to project priorities and limited investigation time, the fix was not finalized and the issue was ultimately deferred, with plans to revert to Rev C hardware and move further investigation to the backlog.

Details:

Observed:

  1. Rev B operated normally.
  2. Rev D exhibited:
    • Heavy frame loss
    • Increased latency
    • Jitter
    • Ethernet link instability
  3. EXFO testing showed:
    • RX clock frequency not detected, or
    • Frequency offset exceeding 150 ppm
  4. RJ-45 port repeatedly linked up and down on Rev D.
  5. SFP port generally behaved normally.

Troubleshooting:

Initial Clock Investigation

Suspected that the new MEMS oscillator was causing timing instability.

Checks included:

  • Comparing oscillator specifications
    • Quartz: 20 ppm
    • MEMS: 25 ppm
  • Measured oscillator amplitude
  • Added resistor divider to reduce clock amplitude

Result: No improvement.

Reset Sequencing Investigation

Hypothesis:

  • IP175 switch chip datasheet recommends holding the device in reset before applying an external clock.

Actions:

  • Added firmware delay before releasing reset.
  • Increased delay to 100 ms.

Result: No improvement.

Hardware Modifications

Tested:

  • Added 15 pF capacitor on oscillator output.
  • Removed ferrite bead from oscillator power supply.

Result: No improvement.

Oscilloscope Analysis

Further investigation focused on clock quality.

Observations:

  • MEMS oscillator waveform appeared distorted.
  • Similar distortion observed on UbiSwitch hardware.
  • Later determined some oscilloscope observations were likely measurement artifacts (“red herring”).

Despite this, all evidence continued to point toward the clock source.

External Clock Testing

A 25 MHz signal generator was connected in place of the onboard oscillator.

Tests included:

  • Varying clock amplitude.
  • Varying frequency.

Result:

  • Issue disappeared.
  • Ethernet performance returned to normal.

This strongly implicated the onboard clock source.

Alternative Oscillator Testing

Several replacement oscillator types were evaluated:

  • Alternative MEMS oscillator.
  • Quartz crystal.
  • Silicon oscillator.

Best result:

  • LMK6CE02500CDLFR silicon oscillator

Result:

  • Issue completely resolved during lab testing.

What Worked:

  • Rev B hardware
  • External 25 MHz function generator clock
  • LMK6CE02500CDLFR silicon oscillator

What Did Not Work:

  • Original Rev D MEMS oscillator
  • Added reset delays
  • Added output capacitor
  • Removed ferrite bead
  • Clock amplitude damping network

Initial plan:

Move to Rev E hardware.
Replace MEMS oscillator with LMK6CE02500CDLFR.
Repeat pressure testing and validation.
Final Status

Root cause was strongly suspected to be related to the Rev D clock generation implementation, introduced with the MEMS oscillator change.

Evidence:

Rev B worked.
Rev D failed.
External clock fixed the issue.
Replacement silicon oscillator fixed the issue in lab testing.

However:

Subsequent BERT testing reportedly did not show complete resolution from the oscillator change alone.
Root cause was therefore not considered fully closed.
Team elected to revert toward Rev C hardware and place further investigation into the backlog.

Conclusion:

Rev D introduced a clock-related issue that caused severe Ethernet timing degradation under certain test conditions. Multiple fixes were attempted, and replacing the onboard clock source appeared to resolve the issue in lab testing, but later validation prevented the investigation from being fully closed. The issue remains partially unresolved and was deferred in favor of reverting to an earlier hardware revision.

(TS2509–0011)