Re: ISCSI: Why the use of the Urgent Pointer

[Michael Krause <krause@cup.hp.com>]

At 07:22 PM 11/14/00 -0800, Matt Wakeley wrote:
>ronald lee wrote:
>
> > Matt Wakeley wrote:
> > >
> > > Look, this whole discussion comes down to a simple question.  Do you want
> > > iSCSI to succeed as an enterprise storage solution or not?  The 
> intention of
> > > the urgent pointer is to help meet the requirements specified in the 
> iSCSI
> > > Requirements document.
> > >
> > > If we review the iSCSI Requirements document it requires:
> > >
> > > - Low CPU utilization, equal to or better than current technology.
> > >
> > > This requirement cannot be met using off the shelf TCP/IP stacks 
> running in
> > > the OS, especially at 10Gbps. iSCSI solutions will have to be 
> implemented in
> > > adapters that handle the transfer of data, similar to Fibre Channel 
> adapters.
> > > Otherwise, iSCSI will eat up all the CPU cycles performing TCP/IP 
> processing,
> > > and FC will blow iSCSI away.
> >
> > I agree that packet processing needs to be offloaded but this might not be
> > always true.  Some implementations might not care if you dedicate a 
> server to
> > a specific application such as mirroring data across LAN/WAN and you 
> don't use
> > this server for anything else.  This is up to the user.
>
>Still, if the CPU is spinning all it's cycles in the TCP/IP stack, it 
>won't have
>much time left over for mirroring.

It also won't be generating the corresponding workload either.  System 
resources which go well beyond just CPU processing are limited and 
therefore bound the potential workload that a given source can 
generated.  Even if you offload everything, you will still be consuming 
system / memory bus bandwidth which directly impacts the application 
throughput.  Also, when the data is accessed, the D-cache impacts will be 
felt just as if they were not off-loaded (function of how much data is 
actually accessed of the targeted storage block) so the benefits of 
protocol off-load are not always as great as many people contend.

Often much of the benefits have been measured on sub-optimal OS / TCP 
implementations so they seem really large; however when they are repeated 
on an optimal OS / TCP, the benefits are much smaller.  This has always 
been the case and as OS / TCP implementations improve (also benefiting from 
Moore's Law), one needs to constantly verify that the off-load solution 
improve minimally in lock step to maintain sufficient benefit to warrant 
the effort.  This is true for all I/O and not just iSCSI or TOE devices.

BTW, much of the benefit for some new technologies comes more from moving 
to async operations / completions compared to actual copy avoidance / 
off-load especially for small message operations.  For larger messages, if 
all of the data is accessed and the copy operation is performed on the 
target processor making the cache hot, then the copy avoidance did not save 
much and the benefit was derived via the async completion semantics and 
potentially the elimination of a limited amount of user / kernel context 
switch costs.

> >  If someone wants run at
> > 10 GBit rates across WAN, they might need a little more help than just 
> framing.
> > Also, having 100Mbytes of memory is not such a big deal these days with the
> > cost of memory coming down.
>
>If cost is not an issue, why is everyone presuring vendors to reduce 
>prices?  Of
>course cost is an issue. If the extra 100MB of memory is going to add $100 
>to the
>cost of your server, you can bet Sun will be looking around for the cheapest
>solution.  Just like they want to drive every dollar out of the cost of a disk
>drive.

Component cost is only one aspect.  Board congestion, impacts to power / 
thermal, etc. are all aspects that need to be considered in any design 
since these all impact the solution cost.


> >  Instead of having one data connection, if you had
> > 100s of connections, the other connections can still operate since they did
> > not lose any packets.
>
>It wouldn't help if there was 200MB worth of data in the long haul pipe 
>for that one
>connection.

While the long-haul is interesting, many implementations will be designed 
for use within the data center / LAN where the customer value props for 
this technology are extremely strong.  I don't agree with the amount of 
emphasis placed on long-haul support from a specification / requirements 
perspective.  It is important but we really need to see better problem 
statements and cost / benefit analysis put forward to make solid decisions.


>Better to resend only a few packets (the lost ones using SACK) instead of 
>all the
>data that had to be dumped on the floor because the adapter ran out of 
>memory to
>store them.

Measurements of various implementations across various distances (data 
center, LAN, metro, etc.) show 30-40% performance gains for various 
applications including bulk data movement applications when using 
SACK.  Most robust TCP implementations either now do or will quite soon 
support SACK given its clear benefits on the network and the applications.

Note: I do not think we should be constrained to designing a spec to the 
LCD of TCP implementations but should expect that the spec will work 
reasonably well on minimally compliant stacks and will work very well on 
robust compliant stacks.

Mike