Hi, we have an application where we need to fan out one ethernet input into 32 outputs inside a custom chassis. We’re looking at stacking pairs of BB-GGR-C-2’s, but we still need 8 boards chained together to get to 32 outputs? Any suggestion on how to do this more efficiently? Any boards in the works with say 16 output ports? We’d be very interested in that.
Thanks for any help you can provide!
We love questions like this.
“Fanout” isn’t really a term that’s used a lot in ethernet networks. Generally there’s a few reasons why we don’t “fanout” ethernet.
Input and output together
An ethernet port consists of a transmit (tx) and receive (rx) together. In the case of 10BASE-T and 100BASE-T, one pair of wires carries tx, while the other carries rx. In the case of 1000BASE-T (and above), there are four pairs that all carry tx and rx on the said wires (using some sophisticated signal processing to filter them out at either end). So, unlike in fanning out digital signals, fanning out ethernet signals means fanning out both an input and output signal.
Bandwidth limitations
Ethernet ports have a stated maximum bandwidth. Once you exceed that bandwidth, you’ll start to lose data. This means you have to be careful with “fanning out”. Typically in an ethernet switch, you might have some ports designated as downstream ports (to connect to devices), and then you’ll have a single upling port that goes to the next part of the network (could be a router, a computer, or another switch).
However, if all your downstream ports are 1000BASE-T, and your upstream port is also 1000BASE-T, you can imagine there are problems if all four downstream ports are trying to send at 1000Mbps into the single upstream port. Since the upstream port is only 1000BASE-T, then four ports cannot send 4 streams of 1000Mbps into a 1000Mbps capable port simultaneously. What will happen in this case is that the switch will attempt to buffer (store) some of the data from the four ports and wait for the upstream port to be free, however eventually that internal switch buffer will overflow and data will be lost.
How do you actually achieve “fanout”? - Use a higher bandwidth upstream port
One way of solving this potential problem is to ensure the upstream port has a higher bandwidth than all the downstream ports combined. For example, you could use a 5GBASE-T or 10GBASE-T for the upstream port, and never see this issue. Of course, that means you need to select a switch that has a higher bandwidth port. On our products, only UbiSwitch has this capability.
This approach doesn’t scale well. If that upstream port feeds into the downstream port of another switch, then that other switch suddenly needs to ensure that its upstream port can handle the traffic from all its downstream ports.
How do you actually achieve “fanout”? - A real understanding of bandwidth
The assumptions above are that all ports will be sending at full nominal data rate simultaneously. This basically never happens in a network, which is why you can get away with daisy-chaining switches together.
However, the problem gets a bit more complex the more daisy chain connections you make. You can end up with bottlenecks and an increased likelihood of dropped packets. Take a look at a single “fanout” below.
Speaking in terms of theoretical maximum, you’d need no more than 500Mbps on each downstream port of Switch A. (This assumes that both downstream ports are sending directly into the upstream port, which not actually be the case in a real world application).
Given that limitation, you’d need no more than 500Mbps coming from the upstream ports of Switch B and C, meaning the downstream ports are essentially limited to 125Mbps maximum.
All this assumes theoretical maximums (you won’t actually get 1000Mbps on a 1GBASE-T port, it will be more like 948Mbps when accounting for packet overheads). This also assumes the maximum throughput, and generally it’s never good to run a network like that. In reality, your maximum constant throughput on the downstream ports of switches B and C is going to be lower than 125Mbps.
This also assumes that it’s just a constant flow of data, but that’s generally not how data flows on a network. You may find ports send bursts of data periodically at a higher rate, then nothing for a while. All this makes it really hard to calculate whether your daisy chain approach is going to work. You can use tecniques like VLAN and QOS to segregate and control network flows, but this requires experience in networking that I don’t believe you possess at the moment (no offence, and feel free to correct me if I’m wrong).
In reality, if you need 32 ports, daisy chaining switches like this to achieve a “fanout” is not a practical solution. You’re better off using a switch with a higher port count. A single switch chip uses a “non-blocking fabric”, meaning it can handle all data rates simultaneously on all ports, in the silicon. So much easier to find a higher port count switch (if you can). You also have to check the internal structure of that higher port count switch. Some actually use a daisy chain archtecture internally which means you’re back to the problem described above!
Can I suggest a different approach…
Use four UbiSwitch and BaseBoards. With this, you can have 8 ports easily on each UbiSwitch, and then a higher bandwidth 10Gbps backplane to stitch them all together. That gives you 32 1Gbps ports. There’s still a likelihood of bottlenecks, but you’ve reduced that by reducing the number of fanouts/daisy chains you need to make, and ensuring the daisy chain connections have plenty of bandwidth to handle 8Gbps and then some.
This approach is about $300 more expensive (calculation below), but it’s far more likely to actually work.
8 x BB-GGR-C-1 = $2616
4 BB-UBS-B-1 + 4 BB-UD1-B-1 = $2952
You could also use a 10G SFP and connect that to the spare 10G port on each UbiSwitch to get an extra ninth 1G port per UbiSwitch.
Let me know if you have any questions.
Thanks so much for the detailed response! That’s very useful. One key aspect of our application (that I should have included initially) is that we have very LOW bandwidth requirements ~ 5 Mbit/s total. We’re broadcasting control messages out to a bunch of embedded PIC32 microprocessors. And in some cases getting health/status type messages back to our central server - but those messages are small too. So nothing that will push the overall bandwidth limits. Physical size is the biggest requirement… we’re putting this in a custom chassis and space is at a premium. Given that, would you suggest the ubiswitch/baseboard approach is the way to go? Could you provide a link to CAD models so our ME team could figure out a way fit things in the tight space? Oh, and we do need NDAA compliant parts – awesome that option is available. Thanks!
That makes sense, if the real data rates are really that low then you should be ok. How tight is tight, in your application; how much space do you have?
It’s still more elegant to use UbiSwitch and do less cascading. It’s generally not super advisable to have multiple levels of cascaded switches.
Links to CAD for all our boards can be found at the link below.
If the 32 devices are all using 100Base-T, will the SFP+ ports still be able to provide this low of speeds to the PicoClasp ports, or will they need to use 1GBase-T to function correctly?
It really depends on the amount of traffic flowing on the 32 100BASE-T ports. If it’s low enough, you shouldn’t see a bottle neck if you use the SFP interconnects at 100Base-T.
However, why not use 1GBASE-T (or even 10GBASE-T) on the SFPs for the UbiSwitch interconnections? It can’t hurt to have a higher bandwidth on the switch interconnects. Or do you have some kind of other limitation here?
