RE: switch latency (was Command Queue Depth, Asymmetric/Symmetric)

["Douglas Otis" <dotis@sanlight.net>]

Merhar,

FIFO depth comes into play as an additional factor.  If an average port
includes somewhere from 16-64 packet depth for asynchronous traffic, these
points of buffering become significant.  Expectation of delay should be in
milli-seconds as few paths are limited to just a few such points of
buffering.  3 micro-seconds per mile in glass provides 150 micro-seconds to
travel 50 miles.  If the lowest level switch is 1 Gb, then there is about
1.5 micro-seconds per 1.5k packet buffered.  Should a buffer depth be 10 at
that point in time, then one switch may add an additional 15 micro-seconds.
Once data is delivered into the NIC, even more significant buffering takes
place.  Although it may be possible to consider micro-second delivery within
a Metro-Area-Network, drop just one packet and this will more than triple
transit time.  If you expect a level of performance, you should consider
both distance, buffering and head of queue blocking.

Doug

> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> Merhar, Milan
> Sent: Friday, September 15, 2000 8:35 AM
> To: 'ips@ece.cmu.edu'
> Subject: RE: switch latency (was Command Queue Depth,
> Asymmetric/Symmetric)
>
>
> Michael,
>
> You're quite right about the actual crossbars introducing
> minimal delays - typically, only a few tens of bit times
> at 2-3x the port bit rate.
>
> But, there's often a store-and-forward delay at the input port,
> where the first bits received sit until the last bits of the
> packet arrive (introducing one packet transmission time's delay,
> or about 12.5 usec for a 1500 byte Gigabit Ethernet packet.)
> This is also where packet inspection and classification
> delays can appear, as you correctly point out.
>
> There is a similar store-and-forward delay coming out of
> the crossbar, where the first bits switched may wait until
> the last bits clear the crossbar, before the actual
> packet transmission is scheduled to the outgoing port.
> As the crossbar may run at 2-3 times the input bit rate,
> this delay is less, often on the order of 4-6 usec for
> a maximum-size packet.
>
> Added together, this gives you 16-20 usec "first bit in to first
> bit out" delay for a maximum sized Gigabit Ethernet packet,
> which would be reported in the usual data-sheet convention
> as about 5 usec "last bit in to first bit out."
> Some of the big switches introduce a little more store and
> forward delay, as they use two levels of internal switching,
> one at the module level, and another at the chassis level.
>
> "Sub 1 usec" numbers may indicate the switch selectively
> uses cut-through forwarding rather than store-and-forward
> (tricky to get right, but certainly faster,) or may merely be
> the result of doing the measurements with a minimum-sized packet.
>
> As for delay numbers exceeding 1 millisecond, all I can suggest
> is that someone was looking at the specs for a 10 Megabit
> Ethernet product (where an 1518 byte packet takes 1.2 msec
> to transmit,) or for a software-based bridge or router,
> which can easily build up 1 msec of queuing time.
>
> And, of course, when you start considering large WAN networks,
> with multiple router hops and substantial speed-of-light delays,
> 30-50 msec starts to look mighty fast.  Add in congestion queuing,
> and you can easily build up to the often quoted 150-300 msec
> "Internet latency."
>
> - milan
>
> -----Original Message-----
> From: Michael Krause [mailto:krause@cup.hp.com]
> Sent: Tuesday, September 12, 2000 11:21 AM
> To: Scott Bradner
> Cc: ips@ece.cmu.edu
> Subject: Re: Command Queue Depth (was asymmetric/Symmetric)
>
>
> At 07:14 AM 9/8/00 -0400, Scott Bradner wrote:
> > > I don't know what the
> > > latency through ethernet switches is, but I'd hope it wasn't
> in the ms
> > range.
> >
> >processing time is in the 10 to 50 microsecond range
>
> Don't know whose switches you are evaluating.  They range from sub 1-usec
> to perhaps 5-10 usec - it is all a matter of how much additional
> processing
> is going on within the switch.  For example, many switches
> examine not just
> the ethernet headers/tags but also look at the IP headers and provide
> additional functionality based on their content.  This is where
> many switch
> vendors try to differentiate and is not a function of whether they can
> switch fast or not. The actual switching elements are usually crossbars
> that operate in the 100-500ns range - chip process improvements
> will lower
> that by quite a bit over the next couple of years.
>
> Mike
>