DIGITAL DATA PROCESSING METHODS AND APPARATUS FOR FAULT ISOLA ICN Reference to Related Applications

This application is a continuation in part of United States Patent Application Serial No. 08/309,210, filed September 20, 1994, the teachings of which are incorporated herein by reference.

Reservation of Copyright

The disclosure of this patent document contains material which is subject to copyright protection. The owner thereof has no objection to facsimile reproduction by anyone of the patent document or of the patent disclosure, as it appears in the United States Patent and Trademark Office patent file or records, but otherwise reserves all rights under copyright law.

Background of the Invention

The invention disclosed and claimed herein relates to digital data processing and, more particularly, to methods and apparatus for fault detection and isolation.

A shortcoming of many conventional digital data processors is their inability to detect and isolate a range of data transfer and operational faults. Personal computers and workstations, for example, are typically configured to detect only a single class of data transfer faults, such as, parity errors. When detected, such a fault can cause the computer or workstation to halt operation, or "crash." Larger computer systems often incorporate error-correcting codes to detect and correct single-bit errors. These systems can be equipped to continue operation even in the event of fault.

Computer systems marketed and described in prior patents to the assignee hereof are capable of both detecting, correcting, and continuing operation after faults. U.S. Patent No. 4,750,177, for example, discloses a fault-tolerant digital data processor having a first functional unit, such as a central processor, with duplicate processing sections coupled to a common system bus for identically processing signals received from other functional units,

such as the memory or peripheral device controller. In addition to checking the received data, the first functional unit compares the output generated by the sections while one of them — the so-called "drive" section - transmits processed data to the bus. When the section outputs do not agree, the functional unit drives an error signal onto the bus and takes itself off-line. According to the patent, a functional unit such as a central processor can have a redundant partner that is constructed and operates identically to the original. In such a configuration, if one of the partners is taken off-line due to error, processing continues with the partner.

According to U.S. Patent No. 4,931,922, also assigned to the assignee hereof, redundant peripheral control units, each with duplicate processing sections, control the latching of data and signaling of errors vis-a-vis data transfers with attached peripheral devices. For this purpose, data signals applied to the peripheral device bus by either the control units or peripheral devices are captured and compared by processing sections within each control unit. The results of those comparisons are shared between the controllers. If the comparisons indicate that the data captured by both control units agrees, then control units generate a "strobe" signal that causes the data to be latched. If the units do not agree after successive retries, the control units withhold issuance of the "strobe" signal and enter an error-handling state for determining the source of the error.

Co-pending, commonly assigned U.S. Patent Application 07/926,857 discloses, in one aspect, a digital data processing system having dual, redundant processor elements, each including dual, redundant processing sections that operate in lock-step synchronism. Failure detection stages in each section assemble signals generated by that section into first/second and third/fourth groups, respectively. Normally, signals in the first group match those of the third group, while the signals in the second group match those of the fourth group. The groups are routed between the sections along conductor sets separate from the system bus. To detect fault, each stage compares a respective one of the groups with its counterpart. If either stage detects a mismatch, it signals an error to its respective processing section, which can take the corresponding processor element off-line.

While the above-mentioned patents and patent applications describe systems with high degrees of fault detection and fault tolerance, such capabilities are not required for all digital

data processing systems. While fault tolerance can be sacrificed in more moderately priced systems, fault detection cannot be. Indeed, in systems that lacking fault tolerance, it is desirable not only to detect fault, but to isolate its source.

An object of this invention, therefore, is to provide digital data processing apparatus and methods with improved fault-detecting capabilities.

A related object is to provide such apparatus and methods with fault isolation capabilities, i.e., that of detecting functional units that are the sources of faults and taking those units off¬ line.

Further objects of the invention are to provide such apparatus and methods that can be readily implemented without excessive cost.

A related object is to provide such apparatus and methods as can be implemented in high¬ speed hardware elements, such as ASIC's.

Summary of the Invention

The aforementioned objects are among those met by the invention, which provides in one aspect fault-isolating digital data processing apparatus including plural functional units that are interconnected for point-to-point communications by a plurality of buses. A system with two central processing units (CPU's) and two input output (I/O) interface units, for example, can employ six bidirectional buses to couple each unit to the others. The functional units monitor the buses to which they are attached and signal the other units in the event there are bus communication errors. Thus, for example, if one of the functional units detects a parity or error correcting code (ECC) error during the transmission of data, that unit can signal a bus error the other units.

The functional units can simultaneously enter into an error isolation phase, e.g. , in response to a bus error signalled by one of the units. During this phase, each unit transmits test data (e.g., predetemined patterns of O's and l 's) onto at least one of its attached buses. Though more than one unit can transmit test data at a time, only one unit can do so on a given bus. For example, in a system with two CPU's and two I/O interface units, both CPU's can simultaneously drive test data over the buses that couple them to the interface units. Later in the phase, both I O units can drive test patterns to the CPU's back over those same buses.

The functional units continue to monitor the buses and to signal bus errors while the test data is being transmitted. Each unit does this not only when another is transmitting test data on a common, point-to-point bus, but also when the unit itself is driving such test data. The latter affords each unit the opportunity to perform a loopback comparison to match the data that it intended to drive onto a bus with that actually "received" from the bus through monitoring. For example, in a system as described above, each CPU monitors its buses for error while the I/O units are driving test data and, in addition, while the CPU itself is driving such data. Moreover, each CPU compares the test data that it intended to drive on the bus with that which it receives from the bus.

In addition to signalling bus errors, the functional units can signal unit-level (or "board") faults when they detect fault in their own operation. To this end, each unit includes error isolation functionality that signals a fault based on (i) whether that unit signalled a loopback error with respect to its own operation; (ii) whether that unit or another unit signalled a bus error during the error isolation phase; and/or (iii) whether any other functional unit signaled that it was faulty during the error isolation phase. By way of example, in a system as described above, a CPU can signal the others that it is faulty if it detects a loopback comparison mismatch while driving test data onto the buses. By way of further example, that CPU can signal the others that it is faulty if it and the other functional units detect a bus error during a phase when that CPU drives test data, but not when the I O units drive test data onto that bus. A unit that determines itself fault can, in addition to signalling the other units, take itself off-line.

According to further aspects of the invention, any of the foregoing functional units can include a processing section that generates data for communication to another unit and that includes dual interfaces for applying that data to the associated buses. The interfaces, referred to as the "drive side" and "check side" interfaces, apply to the buses complimentary portions of each datum generated by the processing section. In the foregoing example, a CPU can use dual interface sections to drive one half of each datum generated by its processing section onto the buses.

The drive side and check side interfaces also receive data driven to the buses, whether by that functional unit or another. With respect to data driven by the functional unit, data received by drive and check sides can be matched against the data intended to be driven by the unit for purposes of the loopback comparison. Thus, for example, portions of a word received from the bus by the check side interface can be compared with the portions which had been applied by that interface as part of a "loopback drive check." Likewise, the other portions of a word received from the bus by the check side interface can be compared against those which had been applied by the other interface (i.e., the drive side interface) as part of a "loopback compare check."

Still further aspects of the invention provide methods for fault-isolating digital data processor operation paralleling the processes described above.

These and other aspects of the invention are evident in the drawings and in the description that follows.

Brief Description of the Drawings

A more complete understanding of the invention may be attained by reference to the drawings, in which:

Figure 1 depicts a digital data processing system employing the invention;

Figure 2 depicts the data buses of an "Xbus" according to the invention;

Figure 3 depicts the control buses of an Xbus;

Figure 4 depicts a bit numbering scheme for an Xbus;

Figures 5 and 6 illustrate simple word, read and line write transactions on an Xbus;

Figure 7 depicts the phases of an operation on an Xbus;

Figure 8 depicts the effect of bus BUSYs on the basic bus cycle of an Xbus;

Figure 9 depicts the effect of bus errors on an Xbus;

Figure 10 depicts the basic state machine and state transitions for a bus handler used in connection with an Xbus;

Figure 11 depicts the interconnection between the Xbus and two functional units, or boards, connected therewith;

Figure 12 depicts self -checking parity logic used in connection with an Xbus;

Figure 13 depicts loopback connectivity used in connection with practice of the invention;

Figures 14 and 15 depict the timing and latching associated with the breaking of boards in connection with practice of the invention;

Figure 16 depicts the timing of board breaking in response to loopback errors according to a practice of the invention;

Figure 17 depicts timing of board breaking in response to heuristic or arbitrary breaking in accord with a practice of the invention;

Figure 18 depicts partitioning of the Xbus in accord with a practice of the invention;

Figure 19 depicts circuitry for routing board not broken in connection with practice of the invention;

Figure 20 depicts the routing of information lines in accord with practice of the invention;

Figure 21 depicts routing of 3-way voted lines in connection with practice of the invention; and

Figure 22 depicts error checking logic in accord with practice of the invention.

Detailed Description of the Illustrated Embodiment

Figure 1 depicts a digital data processing system 10 according to one practice of the invention. The system 10 includes multiple functional units, namely, central processing unit (CPU) 12, CPU 14, input/output (I/O) unit 16 and I/O unit 18. Each CPU 12, 14 is coupled to each I O unit 16, 18 via buses 20A - 20D, as illustrated. These buses can include data lines, as well as control lines. CPU 12 is also coupled to CPU 14 via bus 20E, while I O unit 16 is coupled to I/O unit 18 via bus 20F. These buses include control lines, although they can also include data lines. Each functional unit includes a processing section, 12A, 14A, 16A, 18A, respectively, and two interface sections 12B, 12C, 14B, 14C, 16B, 16C, 18B, 18C, respectively, as shown in the drawing. In a preferred embodiment, each functional unit is contained on a printed circuit board that couples to a bus backplane in the system cabinet (not shown) in the conventional manner.

The processing sections 12 A, 14A carry out conventional functions associated with central processing units, e.g., instruction execution. To this end, the processing sections 12A, 14A can include conventional microprocessors or other central execution units (not shown), as well as other conventional elements (e.g., on-board memory) typically employed for those functions. Interface sections 12B, 12C (constituting the drive side and check side interfaces for unit 12) transfer information between associated processing section 12 A and buses 20 A, 20B, 20E which couple CPU 12 with I O units 16, 18 and with CPU 14. Likewise, interface sections 14B, 14C (constituting the drive side and check side interfaces for unit 14) transfer information between processing section 14A and buses 20C, 20D, 20E which couple CPU 14 to those same I/O units 16, 18, and to CPU 12.

I/O units 16, 18 serve as bridges to peripheral device buses 21 A - 21D and, in turn, to peripheral devices 22A - 22D, as illustrated. In the illustrated embodiment, peripheral device buses 21A - 21D operate in accord with the industry standard PCI Bus Specification. Peripheral devices 22A - 22P are also PCI compatible. I/O units 16, 18 can include processing sections 16A, 18A, respectively, for carrying out administrative tasks, if any, required by the I/O units in their roles as PCI bridges. Interface sections 16B, 16C

(constituting the drive and check sides of unit 16) and, 18B, 18C (constituting the drive and check sides of unit 18) provided in each of the I/O units 16, 18, respectively, transfer information between buses 20A - 20D and buses 21 A - 2 ID. This includes converting transferred information to and from the PCI protocol used by buses 21 A - 2 ID. Although each pair of interface sections 16B, 16C and 18B, 18C interface with a respective pair of internal buses 20A, 20D and 20B, 20C, respectively, the sections 16B, 16C, 18B, 18C interface dedicated PCI buses 21A, 2 IB, 21C, 21D, respectively, as illustrated.

Though the illustrated I/O units 16, 18 serve as PCI bridges, those skilled in the art will appreciate that the invention has application to I/O units capable of interfacing and/or controlling peripherals that communicate via any protocol. More generally, it will be appreciated that, although the illustrated functional units are central processing units and I/O units, the invention can be applied to any class of functional units, e.g., central processing units, memory units, I/O units, communications units, etc.

In operation, the CPU's 12, 14 drive information to I/O units 16, 18 over their respective buses 28A - 20D, and vice versa. Each of the units 12 - 18 also monitor the buses to which they are attached and signal the other units in the event that there are bus communication errors. Thus, for example, if CPU 12 detects a parity error or an error correcting code (ECC) error during a transmission of data to I/O units 16, 18 over data lines in buses 20A, 20B, respectively, CPU 12 can signal that bus error to the I/O units over control lines in buses 20A, 20B. CPU 12 can also signal a bus error to CPU 14 over bus 20E. The other functional units 14 - 18 can likewise signal bus errors in the event they detect erroneous transmissions over their respective buses.

In response to a bus error signal by any of the functional units 12 - 18, each functional unit enters into an error isolation phase. During this phase, each of the units 12 - 18 transmits patterns of test data onto its respective buses 20A - 20F while, at the same time, monitoring those buses for error. For example, in response to a bus error signaled by I/O unit 18, CPU 12 can drive test patterns onto buses 20A, 20B, while CPU 14 simultaneously transmits test patterns on buses 20C, 20D. During these transmissions, each of the functional units

12 - 18 monitor the buses 20A - 20D for parity and ECC errors, while CPU's 12, 14 perform loopback checking on their respective buses 20A, 20B and 20C, 20D, respectively. Once the CPU's 12, 14 have completed their test cycles, I/O units 16, 18 take their mrns at driving test data onto the buses.

The functional units continue to monitor their respective buses 20A - 20D and to signal bus errors while the test data is being transmitted. Each unit 12 - 18 does this not only when another unit is transmitting data on a common bus, but also when the unit itself is driving such data. This permits the units to perform loopback comparisons to match the data that it intended to drive onto a bus with that actually received from the bus through monitoring. For example, while CPU 12 is driving test data onto buses 20 A, 20B, it simultaneously compares data values received from those buses to ensure that they are identical with the driven data. As during normal operation, the functional units 12 - 18 signal bus errors to one another in the event that they detect communications faults during the error isolation phase.

In addition to signaling bus errors, each of the functional bus errors 12 - 18 can generate a default (or "broken") if it detects that its own operation is faulty. Thus, whenever a functional unit 12 - 18 detects a loopback error, it will signal the others that it is broken. Furthermore, if any of the units 12 - 18 detect a bus error when it drives test data onto the bus, but not when another unit drives such data, the unit can signal the others that it is broken - so long as none of the other units has, first, signaled broken, e.g., as a result of its own loopback error. By way of example, CPU 12 will signal the other functional units 14 - 18 that it is broken if CPU 12 detects a bus error during a cycle when it is driving test data onto buses 20A, 20B, but not when I/O units 16 - 18 are driving test data onto those buses. On detecting that it is faulty, any of the functional units 12 - 18 can take itself of-line.

A more complete understanding of the construction and operation of the illustrated embodiment may be attained by reference to the section that follows, in which digital data processing system 10 is referred to as "Polo," buses 20 A - 20F are referred to as the "Xbus," buses 21 A - 21D are referred to as the "Ibuses," CPU 12 is referred to as "CPU1,"

CPU 14 is referred to as "CPUO," I/O unit 16 is referred to as "PCI Bridge 1" or "PCIB 1," I/O unit 18 is referred to as "PCI Bridge 0" or "PCIB 0. "

Bus Naming Convention

The Xbus actually consists of 4 data buses and 12 control buses, which are named as described below.

Data Bus Naming Convention

Figure 2 shows a block diagram of the 4 data buses in the Xbus system. The number for the bus is taken from the CPU slot number shown in the illustration; therefore data buses connected to CPU 0 end in 0 and data buses connected to CPU 1 end in 1. The letter for the bus is determined by whether or not the bus is a crisscross bus (i.e. connects an even slot number to an odd slot number) or a straight bus (i.e. connects an even to an even or an odd to an odd slot). Based on this convention, CPU 0 has connections to data bus AO and BO. CPU 1 has connections to data bus Bl and Al . PCI Bridge 0 has connections to data bus AO and Bl. PCI Bridge 1 has connections to BO and Al. In the illustrated embodiment, the PCI bridge cards do not run in lock-step.

Control Bus Naming Convention

Figure 3 shows a block diagram of the control buses in the Xbus system. The control naming convention for the backplane signals uses the signal name followed by the source of the signal and then the destination of the signal. The naming convention for the associated ASIC pins uses the signal name, the direction (in or out), and the connection (n for neighbor, o for opposite, and p for peer). Examples of this naming convention are shown in Figure 3. Table 1 lists the names of all control buses in the Xbus.

Figure 4 shows a description of the bit numbering scheme used on the Xbus.

Terminology

bus cyde-the 24MHz (-41.67 ns) building block from which all Xbus operations are built. A bus cycle is the time which a valid logic level driven by one board on the backplane is seen by all other boards. Two bus cycles compose a bus phase, 4 bus phases compose a bus operation, and one or more bus operations compose a bus transaction.

bus phase-the 12MHz (- 83.3ns, 2 bus cycle) building block from which all bus operations are constructed. There are logically 11 types of bus phases on the Xbus; "Arb", "Info", "Postl " , and "Post2" are the phases that occur during normal operation. When an error is detected, the special phases "Errorl", "CPU test", "CPU Post", "IO Test", "IO Postl", "IO Post2", and "Eπor2" are inserted in the protocol for fault isolation. (The error phases are sometimes collectively referred to as "Post3"). During each bus phase it is possible to transmit two sets of information on a physical set of backplane lines (i.e. "double pumping") though this is not done for all bus phases and/or signals.

bus operation- A bus operation is the basic unit of address and data transmission and checking on the Xbus. It is generally composed of at least 4 phases: an Arb phase followed by Info, Postl and Post2 phases. Bus errors cause the insertion of the error phases after Post2 and increase the number of phases required to complete a bus operation. Bus operations may consist of multiple info transmissions in the case of a block transfer. A bus operation can be thought of as a full one way transfer on the Xbus.

bus sub-operation— A bus sub-operation is an operation initiated by a bus master during subsequent phases of a block transfer. Sub-operations always carry data and BUSYs are ignored. It is generally composed of 4 phases: an Arb phase during which grant inhibit is used, followed by Info, Postl and Post2 phases. Bus errors may increase the number of phases required to complete a bus sub-operation. A sub-operation is differentiated from an operation in that a bus operation for a block transfer consists of the first transfer plus a number of sub-operations consisting of multiple data transfers.

bus transaction—a complete high level exchange of information on the Xbus. Examples include reads and writes. A read transaction between CPU and PCTB is composed of a minimum of two bus operations; one operation provides the address and function code, and one or more operations provide the return data. A write transaction to a PCTB is composed of a minimum of two operations; one operation provides address and function code, and one or more additional operations provide the data.

Figures 5 and 6 illustrate the terminology surrounding simple word read and line write transactions.

Bus Master— A board that has won arbitration. This board drives the info lines in the info phase of the bus operation. A bus master can be a transaction master or a transaction slave.

Bus Slave~A board that has determined the info lines carry information that it must receive. A bus slave can be a transaction master or a transaction slave.

CD different read— A read in which the C and D side ASICs each provide half the data e.g. when reading error status registers. Loopback checking of the bytes driven by the other side is suppressed.

Cyclops~The Xbus to Tbus interface ASIC on the CPU board in the Polo system.

echo transaction— The second half of a peer-to-peer bus transaction between CPUs. Send and echo transactions are not split; grant inhibit is used to ensure that no other transactions occur between the send and echo. This is to prevent re-ordering.

EFQ-Eviction-Flush Queue. This queue exists only on CPU boards. Refer to the Cyclops (Bus Interface) Specification for details.

EFQ-Freeze State-Set via bit 19 of the Bus Interface State Register. This mode only exists on CPU boards. When in this mode, a board will busy all accesses directed to its EFQ except those from its partner unless it is a data return of any size.

Fast Duplexing— Fast duplexing is a form of duplexing in which no memory is updated. This is done when both boards are coming up for the first time as opposed to updating a new board from a running OS.

Freeze-State~Set via bit 14 of the Bus Interface State Register. This mode only exists on boards that can be duplexed. When in this mode, a board will busy all accesses directed to its RWQ except those from its partner.

Gambit -The Xbus to PCI ASIC on the PCI Bridge card. It interfaces to the Xbus and the PCI bus.

I/O virtual address (IOVA)-An IOVA is an address generated by a PCI card. This address is transmitted across the Xbus to the Cyclops ASIC. Inside the cyclops ASIC, the address is translated into a valid system address. The IOVA is used to provide fault tolerance. It guarantees that a PCT card will generate a correct address range.

loopback checking— This is when an ASIC checks that the value it sees on a pin is equal to the value it thinks should be on the pin during normal operation.

loopcheck operation— This is when the bus ASICs drive test patterns of 55/AAs as part of the error protocol in order to determine the site of a fault.

peer to peer transaction— A two part transaction between two CPUs. The Xbus does not have fully interconnected data buses, and transfers between the two CPU boards must occur in two steps: first a send between the CPU and the PCTBs), then an echo from the PCTB(s) to all of the CPU(s). The requesting CPU drives a complete transaction on its A and B buses. The PCTBs look at the address, and determines whether the transaction is directed to the CPUs. If it is, they buffer the transaction in order to repeat the transaction on their A & B buses once Post2 of the last info phase has passed with no bus errors. In this way, both CPUs see the transaction at the same time. Peer to peer transactions between CPU boards require a minimum of four operations.

RWQ—Read/Write Queue. This queue exists only on CPU boards. Refer to the Cyclops (Bus Interface) Specification for details.

Regurgitated Info— A regurgitated info is a Cougar [Conrad, what is Cougar?] generated cycle used during the update process. It is generated by the update-on-line board and transmitted to the update off-line board. It is unique because an update off-line board accepts info cycles even if the base address does not match the base address of the board.

send transaction-The first half of a peer-to-peer bus transaction between CPUs.

single side operation- An operation with data supplied entirely by either the C or D ASIC; e.g. a read from a PCI card. Loopback checking is performed only by the side supplying the data.

TRTD--Transaction identifier. This is a unique binary number used to tie together two bus operations during a split transaction and to identify the source for write transactions (TRTDs on write transactions are strictly for debug). A TRLD is unique only while a given transaction is still outstanding and will be re-used for later transactions by a transaction- master. Note that trid bits 02-00 are used for the slot number of the transaction-master and trid bits 06-04 are generated by an on-board master - thus allowing a board to have 8 unique masters with transactions outstanding. Trid bit 03 is a new bit for Polo that indicates the format of the address; a zero indicates a Jetta style system address is being transmitted and a one indicates an IOVA (I/O Virtual Address) format is on the backplane. The IOVA address needs to be translated into a system address via the map RAM.

Transaction Master-The specific resource on a board that generated a transaction. The transaction master is responsible for generating the TRLD of the transaction.

Transaction Slave— The board on which a transaction was directed towards, i.e. a board in which the function code and address of a bus operation has decoded to.

Xbus Signal Description

The following section describes in detail the various signals that comprise the Xbus. A functional description is included for each type of signal, though not for every individual signal (for example: there is a TRTD field for all 4 buses, however there is only one description that covers both). The following rules were used in creating the signal names:

all lower-case characters the "_" is a delimiter when it is not at the end of a signal name the "_" at the end of the signal name indicates that the signal is low-true a. indicates A bus signals _b indicates B_bus signals

_x_, _y_, and _z_ are used for the three low-true signals on a triplicated net [n:m] is used to describe a multi-bit field

Signal Description

The bused signals are implemented as four point-to-point bidirectional signals. The ability to drive these signals is controlled through an arbitration network. These signals are protected by a single parity bit that accompanies the bus. The buses are interconnected so that each CPU is connected to both PCI bridge boards and each PCI bridge board is connected to both CPUs. Error recovery is accomplished through the XBus error protocol.

The control signals are organized in a set of point to point unidirectional buses. Each of these buses is ECC protected. These buses carry control signals which are not governed through arbitration. Unlike the bused signals, there is a control bus in each direction between every board. This is necessary in order to ensure the single bus view of the system. For example, if one PCI bridge card sees a bus error, that information must be transmitted to all three other boards in order for the boards to all perform the error sequence.

The bused signals and control signals are double pumped at 24MHz each cycle. That is, they carry different data during the first and second 24Mhz bus cycles that make up a single phase. All buses and control signals are active high on the backplane.

A small number of reset and broken related control signals are buffered by the 26S10 transceiver and replicated for three way voting. These signals are active low on the backplane.

The Info Bus

The following signals are collectively referred to as the info bus. Although there are actually 4 sets of these signals (aO,al,bO,bl), for simplicity's sake only the aO version is listed. For example, when the TRID field is described, it should be understood that there are actually 4 TRTD buses: trid aO, trid bO, trid_al, and trid bl.

Table 2. Xbus Bidirectional Buses

signal width description

info_a0[31:0] 128 Xbus info bus - Info is driven during the info phase

(32x4) of a bus operation by the current bus master. This field may contain either an address (physical address or virtual index with function code) or data, depending on the fund op control line.

trid_a0[6:0] 28 Xbus transaction id (TRID) - The trid lines carry the

(7x4) TRTD (transaction ID) during the first cycle of a phase. During the second cycle, it carries the number of phases remaining, the first op bit, and cache coherency bits.

signal width description

func_op_aO_ Xbus func op - This line carries the func_op_ signal which indicates that the current information on the info bus contains a function code that should be decoded. This signal is low true so that an idle bus will indicate that a function needs to be decoded, and thus a no-op function. This bit is valid during an info phase and is protected by parity along with the lower half of the info field. During the second cycle, it is unused (driven to logic 0 on the backplane).

parity_aO Xbus parity - This parity signal covers all of the bidirectional signals on a bus; info, trid, func_op_.

TOTAL 158

The Control Bus

This section describes the control signals. There are actually 12 control buses, but again only one is described here. The names of the twelve control buses are listed in Table 1 , above. For simplicity of documentation, the control bus is identified by bit numbering, similar to the trid field. However, since the meaning of the control bus bits is very significant, each one is described in detail here.

The control buses are protected by a single bit correction, double bit detection ECC code.

Table 3. Control Bus Signals

signal width description

control[0] 12 bus req and ack - During the first half of the phase, this bit

(12x1) is used for bus req. During the second half of the phase, this bit is used for ack.

During the first half of the phase, this bit is driven by a board when it is requesting the bus. As described in later sections, the bus uses a distributed arbitration model loosely based on the Golfbus. Each board in the system drives the bus req and tests all of other boards bus req signals to determine who will drive the info signals during the next phase.

During the second cycle of the phase, this bit is used to acknowledge a bus transaction. Ack is asserted in Post2 by the target board of the transaction. This signal is the result of an address decode, so it is only valid in Post2 of an operation that is transferring an address. Ack provides an indication of whether or not a transaction is progressing. This is relevant in a Polo system, since a PCI card may go away resulting in a no ack for a ping operation.

Acks in Polo system also let a PCI Bridge know that a CPU's map RAM has mapped a PCI initiated access to a valid CPU address. If a PCTB's read or write is not Aed, the PCI slot that initiated the access may or may not be set off-line depending on bits in the Gambit's configuration register.

Writes from the CPU to the PCTB are acked to facilitate debug, but are otherwise unused. A peer to peer CPU write is not ACKed by the PCTBs, but the echoed operation is

control[ 1] 12 grant inh and main int - During the first half of the phase,

(12x1) this bit is used for grant inh, during the second half of the phase, this bit is used for main int.

During the first cycle, the grant inhibit control bit is driven by the current bus master to extend the info cycles when the bus master is moving a block of data. The arbitration logic will not issue a grant to any other board when a board is driving the signal. This ensures that the current bus master will retain ownership of the bus for another cycle.

During the second half of the cycle, this bit is used to signal a maintenance interrupt. Maintenance Interrupt indicates that some board in the system is requesting attention from the software maintenance process. Any board in the system may drive this signal during the second half of a bus phase regardless of bus mastership. All boards in the system will sample maintenance interrupt and use it to reset their arbitration priority.

controlp] 12 bus err a and busy - Assertion of this signal during the

(12x1) first half of Post2 signals that a bus error was detected on the info bus associated with this particular control bus (n,o,p). Any operation in Arb, Info, or Postl will be aborted. Operations in Post2 are suspended while the error protocol runs, and then will return to the Info phase.

The CPU initiator of a peer-to-peer operation must track the entire operation to see whether the cycle is errored in either the send or echo portion of the transaction.

Assertion of this signal during the second half of Post2 indicates to the bus master that the operation should be aborted and re-tried at a later time.

Busy is ignored for the send portion of a peer-to-peer operation. The CPU initiator of a peer-to-peer operation must track the entire operation to see whether the cycle is busied.

control[3] 12 bus err b and funny state - Assertion of this signal during

(12x1) the first half of Post2 signals that a bus error was detected on the B bus connected to this board. Any operation in Arb, Info, or Postl will be aborted. Operations in Post2 are suspended while the error protocol runs, and then will return to the Info phase.

The CPU initiator of a peer-to-peer operation must track the entire operation to see whether the cycle is errored in either the send or echo portion of the transaction.

During the second half of the cycle, this bit is used to signal that a board has just gone unbroken. The board will continue to assert this signal until it has seen eight phases without a bus error occurring. At that point the board will stop asserting this signal. Then all other boards will treat this board as an active, responding board. This prevents a board that is going unbroken from responding to bus errors in the middle of an error sequence that is already underway.

control[7:4] 48 checkbits - The top 4 bits of the control bus are checkbits

(12x4) generated from the lower 4 bits of control signals.

The checkbit algorithm is:

control[4] = control[0] ^A control[l] ^A control[2];

control[5] = control[0] ^A control[l] ^A control[3];

control[6] = control[0] ^A control[2] ^A control[3];

control[7] = control[l] ^A control[2] ^Λ control[3];

TOTAL 108

The Voted Signals

The voted signals are the only signals through 26S10 transceivers. These signals are point to point and terminated at each end, so that insertion of an unpowered board does not disturb the termination (and timing) of a net in use. Only the _x_ versions are listed; there are _y_ and _z_ signals making up the triplet.

Table 4. 3-Wav Voted Signals

signal total description

reset_0_l_x_, 18 reset - There are separate 3-way voted triplets reset_0_2_x_, (2x3x3) from each CPU to the other three boards in the reset_0_3_x_, system. When a RECC needs to reset the reset_l_0_x_, system, all lines go active. When a CPU wants reset_l_2_x_, to reset another board, only the lines going to reset_l_3_x_, that board are active.

board not broken_0_l_x 36 broken status - This three way voted signal is board_not_ broken_0_2_x (4x3x3) driven from each board to each other board. It board not broken_0_3_x is driven when a board is alive and not broken in board not broken l O x the system and is used to determine which buses board not broken_l_2_x are active. The C-side ASICs drive the signals board not broken_l_3_x and the D-side ASICs drive the output enables board not broken_2_0_x for the 26S10s. This organization guarantees board not broken_2_l_x that board not broken is deasserted whenever board_not broken_2_3_x either side of the board thinks that the board is board not broken_3_0_x broken. The receiving board votes the x, y, and board not broken_3_l_x z signals. CPUO in slotO drives board not broken 3 2 x board_not_broken[0], etc.

sync_x_ 3 sync status - These signals are on the CPU only

(1x3) and are used when synchronizing a pair of CPUs to enter the duplexed state.

even_online_x_, 6 on-line status - These signals are on the CPU odd online x (2x3) only and are used by the CPU boards to communicate to each other which CPU board(s) are in the on-line state. Even_online_ is asserted when the CPU in slot 0 is on-line; odd_oniine_ is asserted when CPU in slot 1 is on-line.

signal total description

TOTAL 63

Other Control Signals

Table 5. Miscellaneous Control Signals

lot ida 2 slot id - The slot ID signals are hard wired on the slot idb backplane for each slot. There is one duplicated slot id. Since the slots are dedicated in Polo, it is only necessary to determine if a board is in an even or an odd slot. These two bits will be registered and checked by each ASIC at reset and will not be sampled again. If an error is detected at reset, the board will break and hence will never be capable of being brought on-line.

xb clk8 1 system clock - This is the system clock received by the Sentry clock chip and used to generate the board clocks. It is generated by the backplane clock oscillator. This clock runs at 8MHz, so every board in the system will be in sync with each other and there will be no need for additional synchronization clocks to be passed along the backplane. The clock is pulse width modulated so that 4Mhz can be generated.

slot0_ta_d, 4 ta signals - These "ta" signals are only present on slotO_ta_c_, CPU boards and are sent between a duplexed board slotl_ta_d, pair for early detection of the boards going out of slotl_ta_c_ lockstep.

TOTAL

Xbus Protocol

Overview

The Xbus is a point-to-point synchronous, pipelined, multiplexed address/data, error detecting bus with split-transaction capabilities. Function-code, address and data are parity and loop-back checked. Control lines are ECC protected and loop-back checked. Three-way voting is implemented on the reset, clock, and broken indicator lines.

The bus supports word accesses and has a block transfer capability for support of cache line accesses. The Xbus has a logical 32-bit wide data/address bus.

Bus Operation

The basic component of all Xbus transactions is the operation. An operation is composed of four phases as illustrated in Figure 7: arb, info, postl, and post2. Two information transfers can occur on the bus during each phase; this is referred to as "double pumping" . The double pump frequency is approximately 24MHz. The figure illustrates the logical activity on the bus. All information is directly registered in the ASICs without any external buffers.

The phases are used as follows:

Aarb phase: Boards drive their arbitration request lines during the first half (cycle) of the arbitration phase. During the second half they determine whether they won arbitration and prepare for the info phase.

Info phase: For non-IOVA address transfers, boards drive the virtual index, function code, remote/coherent bits, and byte enables during the first half of the info phase and the physical address during the second half. For IOVA address transfers (IOVA bit in the trid is true), boards drive the IOVA during the first half of the info phase and deterministic data with good parity during the second half; the physical address is

gotten from the I/O address map RAM look-up. For data transfers, data is driven during both the first and second halves of the cycle. Note that non-cache consistent address transfers need not supply a virtual index through the driven information must be deterministic and parity will be computed across it.

Postl phase: During this phase, boards are determining whether any error conditions existed with the info phase and whether there is need to BUSY the operation. CPU boards map the device index portion of the IOVA to obtain the full physical address and virtual index of an I/O board's transfer for IOVA address transfers.

Post2 phase: Any board detecting an error with the info phase drives the error lines during the first half. If a board does get errored, it next goes to the error sequence phases to determine the source of the error. Any board detecting the need to BUSY an address/function code driven during the info phase drives BUSY during the second cycle of this phase. It is also during this phase that accesses are acknowledged.

Bus Busies

Figure 8 illustrates the effect of bus BUSYs on the basic bus cycle. As shown in the figure, BUSY has no effect on a bus operation during any phase except for post2; a BUSY during post2 will cancel the bus operation. Note that busys for multiple cycle bus operations, such as block transfers, have the special rules. Should a cycle be both BUSYed and ERRORed, the ERROR takes precedence.

Bus Errors

Figure 9 shows the effect of bus errors on the basic bus cycle. As shown in that figure, the board that was transmitting during the error automatically gets the first available info cycle following execution of the error protocol. The arbitration is ignored in the previous cycle.

Since the Xbus has no transceivers, the loopcheck phase of the Golfbus error protocol (Post4) has been modified to allow each board an opportunity to verify its transmit and receive capabilities. This has resulted in new states being added to the bus error operation. These states are described below:

Errl: The Errl state is entered on the cycle after a bus error is detected. This state is used to allow for time to mm off the info bus before the loopback checks are performed. A board that is in its info phase during Errl will disable its output enables half way through the phase.

CPUTest: The CPUTest state is used to test the CPU's ability to drive patterns on the Xbus. On the first cycle of the phase the CPU will drive 55 on the info bus, 55 on the trid bus, 1 on the parity line and 0 on the func op line. On the second cycle of the phase the CPU will drive AA on the info bus, 2 A on the trid bus, 0 on the parity line and 1 on the func op line.

CPUPost: The CPUPost state is used to turn the bus around between the CPU's loopback check and the I/O boards loopback check. This phase is also used as a Postl cycle for the CPU's loopback pattern.

IOTest: The IOTest state is used to test the I/O board's ability to drive patterns on the Xbus. On the first cycle of the phase the I/O board will drive 55 on the info bus, 55 on the trid bus, 1 one the parity line and 0 on the func op tine. On the second cycle of the phase the I/O board will drive AA on the info bus, 2 A on the trid bus, 0 on the parity line and 1 on the func_op line. Bus errors from the CPUTest phase are reported during this phase. This information is used to evaluate the bus, CPU, and I/O board at the end of the error sequence. The last I/O ASIC to drive the data bus drives the bus during the IOTest phase.

IOPost: The IOPostl state is used to evaluate the IOTest data.

IOPost: The IOPost2 state is used to transmit any bus errors from the IOPostl state. This information will be used to make an intelligent decision about how to deal with the error.

Err2: The Err2 state is used to evaluate the information from the loopback checks. Bus errors from CPUPost and IOPost2 as well as information shared between the C and D sides of each board are used to determine what course of action to take. This set of actions will be described later in this section.

Figure 10 shows the basic state machine and state transitions for the bus error handler. The key challenge for the bus error algorithm on the Xbus is to diagnose errors so that system operation can continue. Unlike previous systems that use duplicated buses to allow all functional units a guaranteed path for communications, when the Xbus removes a bus from service, it must also remove one of the two units attached to that bus. In some cases, the right thing to do is obvious. In other cases, it is not. The following sections analyze various faults, how they are handled and how they manifest themselves.

Bus Error Broken Conditions

At this point it would be helpful to classify the different types of conditions that cause a board to go broken when a bus is detected bad.

• loopback on control - C and D ASICs must always agree on what to drive on the control lines, including whether or not to assert bus error. If one ASIC asserts bus error and the other side does not, the board breaks.

• loopback on data - C and D ASICs must always agree on what to drive on the duplicated info lines. When driving CD same data, ASICs compare the data they drive with the data they receive. An ASIC asserts bus error on parity errors when receiving data, and parity errors and loopback errors when driving data. Loopbacks checking is disabled when a board drives "CD different" data, such as the contents of error reporting registers or data from PCI cards. The board breaks if the two sides disagree on which bytes have or do not have errors.

• arbitrary - Break the designated board in Err2 when there are bus errors signaled during CPUtest and IOtest and no board has broken by the end of Iopost2. This is called an arbitrary shoot because the fault is most likely on the backplane, so it is arbitrary as to which of the two boards connected to the faulty bus to break. Typically, the CPU is set broken, so that the system can continue with all of its I/O available, but if bit 21 of the Bus Interface State register is set then the PCTB board will be the designated board.

• heuristic - a board breaks itself during Err2 if there is a bus error when it drives, but no bus error when the other board drives, and the other board did not break by the end of IOPOST2.

Xbus Fault Analysis

In order to understand various faults and what they can mean, it is important to present a detailed block diagram of the Xbus interconnect. Figure 11 shows the interconnect for a typical Xbus line.

The black dots in figure 11 represent the connectors to the backplane. For fault tolerance and fault isolation reasons, it is important that the boards should be routed so that the etch between the D-side and the C-side runs through the connector connection. This limits the amount of etch on each board that cannot be isolated to a minimum. On the CPU board, one ASIC both drives and receives a given net while the other ASIC only receives that net. On the I O board, each ASIC can potentially drive every net. The CPU ASICs are always in lockstep and therefore each ASIC is capable of sharing the data out load. However on the I/O board, each ASIC connects to a different PCI bus so a signal ASIC may need to drive the entire Xbus. There are cases in normal operation when only one CPU ASIC will drive the entire bus.

Fault Conditions

The following sections identify all known fault conditions and describe their handling. Refer to Figure 11 to determine the location of the fault site indicated. That figure depicts two functional units (or board), e.g., units 12 and 16, as well as their respective drive and check sides 12B , 12C, 16B, 16C.

CPU Board Faulty Input Circuit - CPU Driving

• fault site 31

• break via loopback on control fault

This fault deals with a fault in the input section of one of the CPU ASICs. In this case, the fault occurred during or just before a cycle in which the CPU drove the info bus. The error is detected when the CPU drives the bus. The ASIC with the faulty circuit will signal a bus error during the Post2 phase of the cycle and the other side ASIC will not. The board will go broken and drive bus error during Errl. The error sequence will be executed, and the operation will be relied by the partner CPU with no error.

CPU Board Faulty Input Circuit - I/O Board Driving

• fault site 31

• break via loopback on control fault

This fault deals with a fault in the input section of one of the CPU ASICs. In this case the fault occurred during or just before a cycle in which the I/O board drove the info bus. If the error is a multi-bit error that evades the parity logic, the error will be caught internal to the CPU board and the CPU board will go broken. If the error is a single bit error the faulty ASIC will detect a bus error during the Postl phase of the transfer. The ASIC will drive bus error during Post2 of the transfer and the other side of the board will not. The board will

break with a loopback on control failure in the next phase. After the error sequence, the operation will be retried by the partner CPU with no error.

CPU Board Different Data C-Side and D-Side

• fault site 32

• break via loopback on data fault

This fault deals with an internal CPU fault that results in different data being driven out of each ASIC. The error is detected when the CPU drives the bus. The C and D ASICs will trade error status and disagree on where the error is during Postl ; both C and D sides will see an error on bytes the other side drives but no error on the bytes it drives. The board will go broken and drive bus error during Post2. The error sequence will be executed and the operation will be retried by the partner CPU with no error.

CPU Board Faulty Output Circuit - Buffer to Pad Fault

• fault site 33

• break via heuristic broken

This is a fault in the output section of the CPU ASIC resulting from an output driver circuit fault that blows in a manner that causes an internal open between the output driver and the ASIC pad, while not disrupting the functionality of the input receiver. All ASICs on the bus will detect a bus error during the Postl phase of the transfer. The ASICs will drive bus error during Post2 of the transfer. All ASICs will detect a bus error during the CPUPost phase. No bus errors are detected during the lopostl phase. The CPU board will go broken during Err2 based on the fact that it detected a bus error when it drove the bus, no bus error when the I/O board drove the bus and the I/O board did not go broken after its error sequence.

CPU Board Open - CPU Board Driving

• fault site 34

• break via loopback on data fault

This fault results from an open due to either a broken etch or a lifted pin on he CPU board. The routing is very important for this class of fault. The etch between the C-side and the D- side should be routed through the connector pin. This limits the possibility that an open on the CPU board is mistaken to be an open on he backplane. In this case the fault occurred during or just before a cycle in which the CPU drove the info bus. The error is detected when the CPU drives the bus. During the Postl phase, the driving ASIC will not see an error but the checking ASIC will signal a compare error. During Post2 the board will go broken and drive bus error. The operation will be retried by the partner CPU with no error after the error sequence.

CPU Board Open - I/O Board Driving

• fault site 34

• break via loopback on control

This fault results from an open due to either a broken etch or a lifted pin on the CPU board. The routing is very important for this class of fault. The etch between the C-side and the D- side should be routed through the connector pin. This limits the possibility that an open on the CPU board is mistaken to be an open on the backplane. In this case the fault occurred during or just before a cycle in which the I O board drove the info bus. The ASIC with the open between it and the connector will detect a bus error during Postl. During Post2 one ASIC will assert bus error and the other will not, causing the board to break. The operation will be retried by the partner CPU without any error after the error sequence.

CPU Board Short

• fault site 34

• break via arbitrary broken

This fault deals with a short on the CPU board. When the fault occurred and who was driving the info bus during the fault are not relevant to this class of fault. During Postl of the transfer all ASICs on the bus will detect a bus error. The ASICs will drive bus error during Post2 of the transfer. Both ASICs on the CPU will detect a loopback error during the CPUPost phase and the ASICs on the I/O board will signal a bus error during IOTest. The ASICs will detect a bus error during the lopostl phase and signal a bus error during IOPost2. During Err2 the designated board will go broken based on the fact that it has detected bus errors during the error sequence and no other boards went broken after the error sequence.

Backplane Open Etch

• fault site 35

• break via arbitrary broken

When the fault occurred and who was driving the info bus during the fault are not relevant issues for this class of fault. During Postl of the transfer some ASICs on the bus will detect a bus error and drive bus error during Post2. Both ASICs on the I/O board will detect a bus error during CPUPost. The CPU ASICs will detect a bus error during IOpostl. During Err 2, the designated board will go broken based on the fact that it has detected bus errors during the error sequence and no other board went broken during that error sequence.

Backplane Short

• fault site 35

• break via arbitrary broken

When the fault occurred and who was driving the info bus during the fault are not relevant issues for this class of fault. All ASICs on the bus will detect a bus error during Postl and drive bus error during Post2. All ASICs on the bus will detect a bus error during CPUPost and IOPostl. During Err2, the designated board will go broken based on the fact that it has detected bus errors during the error sequence and no other board went broken during the error sequence.

I/O Board Faulty Input Circuit

• fault site 36

• break via loopback on control

This fault deals with a fault in the input section of one of the I/O Board ASICs. For this particular fault, it is irrelevant who was driving the backplane when the fault was detected. The faulty ASIC will detect a bus error during Postl . During Post2 of the transfer, the faulty ASIC will drive bus eπor and the other ASIC will not, causing the board to go broken. If it was a CPU initiated request the operation win be retried by the CPU with no error after the error sequence. If it was a request initiated by the I/O board, then the request will be dropped.

I/O Board Output Circuit Fault - Buffer to Pad

• fault site 37

• break via heuristic broken

This is a fault in the output section of the I/O board ASIC. This class of fault results from an output driver circuit fault that blows in a manner that causes an internal open between the output driver and the ASIC pad, while not disrupting the functionality of the input receiver. All ASICs on the bus will detect a bus error during Postl of the transfer and drive bus error during Post2. No error is detected during the CPUPost phase. The I/O ASICs will detect a bus error during IOpostl. During Err2 the I/O board will go broken based on the fact that it

has detected a bus error when it drives the bus, no bus error when the CPU board drove the bus and the CPU board did not go broken after its error sequence.

I/O Single-side Access - ASIC Parity Gen. Fault

• fault site 38

• break via internal parity generator broken

A fault in the parity generation logic during single side accesses (data driven entirely by either the C or D ASIC) could cause the system to bus error forever. The otherside ASIC doesn't know the data, and has no way of checking the parity. For this reason, the info bus parity generation is duplicated and selfchecking inside of the Gambit ASICs.

I/O Single-side Access - ASIC PCI Data Path Fault

• fault site 38

• break each PCI slot due to checksum and may RAM errors

This is a fault within the ASIC's PCI data path. This hardware is not running in lockstep with the other side of the board and therefore is not self-checking. Eventually bad addresses produced by the PCTB will cause map RAM errors and/or data checksum errors in the CPU ASIC. The CPU ASIC noACKs accesses that cause map RAM errors, and the PCTB sets off-line the PCI slot that originated the noACKed cycle based on an option bit in the Configuration register in page zero of SAM compatible I/O space. Eventually, all PCI slots handled by the detective ASIC will be set broke. In the meantime corrupted I/O data is detected by checksums and handled by higher level protocols.

I/O Non-single-side Access, Different C-D Data

• fault site 38

• break via loopback on data

This fault deals with an internal I/O board fault that results in different data being driven out of each Gambit ASIC during a regular (not single-side) read return. Each ASIC will disagree with the data driven by the other side and the board will break with a loopback on data error. The error sequence will be executed and the operation will be retried by the CPU and get noACKed.

I/O Board Open

• fault site 39

• break via loopback on control

This fault deals with an open, either from a broken etch or a lifted pin on the I/O board. The routing is very important for this class of fault. The etch between the C-side and the D- side should be routed through the connector pin. This limits the possibility that an open on the I/O board is mistaken to be an open on the backplane. The driving ASIC sees no errors and the outside ASIC asserts bus error; the board breaks the following phase with a loopback on control fault. After the error sequence the operation will be retried with no error and get noACKed.

I/O Board Short

• fault site 39

• break via arbitrary broken

This fault deals with a short on the I/O board. When the fault occurred and who was driving the info bus during the fault are not relevant issues for this class of fault. During Postl of the transfer all ASICs on the bus will detect a bus error and then drive bus error during Post2. All ASICs on the bus signal a bus error during CPUpost and IOpostl . During Err2, the designated board will go broken based on the fact that it has detected bus errors during the error sequence and the other board did not go broken after its error sequence.

Transient Fault

• fault site anywhere

• action depending on fault site and timing

This fault deals with a transient fault. A transient fault is defined as a fault that is detected, but cannot be reproduced by the error sequence. If the fault is on the driving board and it is caught by loopback check, that board will go broken. If this is not the case then no board will break until Err2, then the designated board will go broken.

Slow Driver Fault

• fault site 34, 39

• possibly break the wrong board with a loopback on control fault

When an ASIC with a marginally slow driver drives the bus, four ASICs clock in data from the net while the net is changing. Due to differences in speed between ASICs from different lots, its possible that some of the ASICs will detect a bus error, and some won't, resulting in a loopback on control fault. This may result in one or both of the boards on the bus going broken.

Board Not Broken Generation

The board_not_broken_out signal is generated by all boards. If a board is going to break the normal operation is for the board to assert bus error. All boards will enter the error sequence and the board that was going to break will de-assert board_not_broken_out during the error sequence. It is possible that a board could break without asserting bus error. This could only happen if there is a problem with asserting bus error on the d-side gate array (for instance the clock line going to the flip-flop that produces the control signal (bus error) opens).

Arbitrary Breaking

There are a series for cases mentioned before that error logic cannot determine where the fault is. For these cases the goal is to not use the bus that the two boards are connected to. The default is to break the CPU board (designated board). This makes sense because there is a partner running in lock-step so no connectivity is lost. It is possible that the CPU board will break but the fault really lies with the PCTB. The only way this will pose a problem is if the failure in the PCTB effects the other bus that is still connected and is being used. Normally it is the CPU board that is the designated board.

Xbus Fault Tolerance

The goal for fault tolerance on the Xbus is for no single point of failure to cause a system crash or allow a transaction to complete without transmitting the correct data. A single point of failure could be any component, any signal, or any pin on a component.

The Xbus fault tolerance scheme assumes that one of the components connected to the bus is always responsible for hard bus errors. To this end, the design has been simplified around the assumption that when an error occurs, an offending board, or Field Replaceable Unit (FRU), will be removed from service. This extends to include all buses that the FRU is connected. Thus when an Xbus fault occurs the faulted bus service is removed taking at least one FRU with it.

Both sides of the CPU board drive signals to the bus and both sides of the board perform various checks to ensure that the board is functioning normally. This is identical to the Golfbus methodology. The PCTB board behaves differently; both sides of the board drive and check the bus when driving data from Xbus related IO registers, and when driving data originating on a simplexed PCI card, only one ASIC drives and checks the entire width of the bus.

Info Bus Protection

The info bus is protected by parity and loopback checking. However there is no A to B bus cross checking by anyone on the bus other than the bus master. When a PCTB is bus master only one of the two buses attached to a given CPU module will be active because the PCTBs do not duplex and therefore the bus from the PCTB that did not win arbitration must be idle. Likewise the PCTBs cannot know if two duplexed CPU modules are driving the bus or a single simplexed CPU has ownership (two simplexed CPUs could have started arbitration at the same instant). Thus receivers on the Xbus will only listen to the bus that is driven by the module that won arbitration.

Parity

Parity is generated and transmitted across each of the physical buses. The 32 bit bus, trid field, and funcop bits are all covered by a single parity bit. The purpose of the parity check is to protect against etches that open between the transmitter and receiver. Boards in the system check the transmitted parity that they receive with parity that they generate themselves.

A bus error is signaled on any parity error. The bus en signals are also effected by other error checking that is described later in this document. In an effort to simplify these descriptions the reader may assume that the actual error signals are a logical OR' ing of the various outputs from different checking blocks.

Both the C and the D-sides of the boards will perform the check and compare results individually and decide on any action that must be taken. In the event that the two sides of a board do not agree upon the statute of a bus cycle, the board will break in the following cycle in one of two ways.

• The D-side of the board detected an error and the C-side did not. In this case the D- side will drive a bus error and the C-side will disagree with what the D-side has done and break the board. Here the transaction is errored and repeated, but the broken board should no longer be checking and the transaction will complete.

• The C-side of the board detected an error and the D-side did not. Here the C-side will once again break the board, but this time because it expected to see an error on the bus and didn't.

The transaction will complete normally and the bad board will be removed from service.

CPU boards compute parity in both C and D ASICs and break if a defect arises in one side's parity generators. Most of the transfers from a PCTB board, however, originate from a PCI bus connected to a single Gambit ASIC. When the Gambit drives this single sided data, it

drives all the bits on the Xbus, and an error in the simplexed parity generator would hang the system. Fortunately, there is no need to duplicate the entire Gambit ASIC, only the parity generation section.

Referring to Figure 12, a fault at site 41 causes erroneous data to be transmitted with correct parity. This will be detected by higher level checksums. A fault at site 42 will cause the board to break with a data loopback fault. A fault at site 43 will cause the board to break with a parity generator fault. A fault at site 44 or 45 will break the board with either a data loopback fault or a parity generator fault.

Loopback Checking

The loopback correctivity is shown in Figure 13. On normal accesses (non-CD different, non-single sided), each side of the board drives half of each field to the bus and receives all of both busses. Thus there are two checks that each ASIC can do on the data returning from the bus.

1: Does the returning data match the data that was driven from this ASIC. This will be known as the loopback drive check.

2: Does the returning data match the data that was driven by the other ASIC. This will be known as the loopback compare check.

Two error signals are generated inside of each ASIC for each physical bus that it is driving. In order to know the true state of the board each side of the board must know the results of the comparisons on the other side as well as the comparisons it has done. Thus two signals are passed in each direction, as shown in figure 13, so that the each side may correctly determine the state of the board.

Each ASIC will determine whether the data on the bus is faulted or not, as well as if this board is broken or not. Table 4 below shows what combinations of the loopback signals for each bus indicate that the board is broken.

B x n l B

A double bus error will be asserted and the not broken signal de-asserted if a condition exists that should break the board. In addition a bus error will be asserted for any condition where a compare error of any type has occurred and the board has not broken. The bus error will only be on the bus while the error is detected.

Loopback on CD Different Accesses

When an ASIC drives data from a CD different register on the bus, it performs loopback checking only on the trid, func_op_, and parity lines. It asserts bus error if it sees an error, and will go broken according to table 4.

Loopback on Single-side Accesses

When a Gambit ASIC drives single-side data from the PCI it performs loopback checking on all of the bits; it does this only for the purpose of asserting bus error, not to see if it should go broken. It will not break according to table 4, because the cmp_err and drive_en signals from the otherside are invalid. However, if the driving ASIC sees a bus error, and the otherside ASIC does not (parity received good), the board will break due to a loopback on control fault.

Loop ck ops

Loop ck ops are used during the error sequence to make intelligent choices about the state of various boards and the buses that they are attached to. A loop check op is a cycle in which a board drives an alternating pattern of AA's and 55 's on the backplane. The board evaluates the pattern and similar patterns driven by boards on the other end of the bus. Based on this information, bus state is altered.

Control Bus Protection

The Xbus collects control signals into a series of point-to-point unidirectional signals protected by an ECC code. Each board drives a separate control bus to each of the other boards.

There are two completely independent types of checks performed:

1: Every ASIC corrects for any single point failures (shorts, opens, dead drivers or receivers) with a single error correcting, double error detecting ECC code. Single bit failures cause a maintenance interrupt and are logged, but otherwise have no effect.

2: If a driving board detects a double bit error on the outgoing control signals it will attempt to assert bus error and then it will break in the enor sequence (it will break regardless of its ability to drive bus error). If a receiving board detects a double bit error on any incoming control signals it will attempt to assert bus error then it will break in the enor sequence. If the enor was actually on one board and only one board saw the error then the correct board will break. If the error was such that both boards saw it then both boards will break. This is to limit the possibility that data gets corrupted.

Three-way Voted Signal Protection

Some broken and reset related control signals on the Xbus are triplicated and voted. The reset and board_not_broken_out signals are buffered through transceivers because they must be valid during power up or power down events. The CPU signal sync out is buffered through transceivers to reduce ASIC pin count. The CPU signal my online out is sourced directly by the ASIC as a cost reduction.

The D-side ASIC drives signals to the transceivers and to the C-side ASIC. The C-side of the board checks what was driven by the D-side with what it thinks should have been driven. If the C-side disagrees with the D-side the board breaks.

Each ASIC that receives a three-way voted signal will perform a majority voting algorithm on the three signals to determine the value. In normal operation all three lines will be the same. If there is a fault on one of the three lines (either on the backplane or on the board), two of the three lines still will agree.

When a voting error occurs, the device detecting the fault should send a maintenance interrupt and log which transceiver has the non-unanimous vote. The logged information is stored in the Vote Error Transceiver Status register(s) and will not be over-written until the status register is explicitly re-enabled by software.

Error Reporting

A number of errors can be detected by a board's bus interface logic:

1: A loopback error that breaks the board. This specifically refers to loopback errors on the info, parity, TRTD, and func op signals which are seen by one side of the board and not the other during bus cycles in which the board drove the bus.

2: A disagreement between the two-sides of a board on whether a three-way voted signal should be driven.

3: A parity error on the bus, either on a cycle where the local board was driving the bus or another board was driving the bus.

4: A loopback error on the bus.

5: A three way voter enor in which not all three signals are in agreement.

6: Any bus error signaled by any board in the system.

7: A thermal fault (high temperature detected on the board).

Some of the above errors may break a board (e.g. #31 and 32), some may cause a bus error to be generated (e.g. #33,34 and 36), and some have no effect on normal machine operation (e.g. #37). The Xbus interface contains two classes of stams registers to record information surrounding the above errors: Broken Stams and Error Stams. Broken Stams registers are only activated when a board goes broken, and are designed to pinpoint the logic that set broken. Error Stams registers are for reporting non-fatal errors such as faults on the backplane. All of the Error Stams registers are updated simultaneously on any bus error signaled by the system, or a voter or thermal enor on the board. Specifically, when all error reporting is enabled, any of the above errors cause:

1 : A maintenance interrupt to be generated. 2: The type of error condition to be stored.

In the above case, subsequent error state recording and the resulting maintenance interrupts are disabled until explicitly re-enabled.

It is not always optimal that all the above errors trigger the error registers and generate a maintenance interrupt. The reporting of new errors can be effectively blocked by an excessive number of reports on a known error. Hence, it is possible to torn off error latching due to different sources:

1: Perform error latching and maintenance interrupts on all errors.

2: Inhibit error latching and maintenance interrupts of incoming three-way voter . errors.

3: Inhibit error latching and maintenance interrupts on thermal failure related errors.

The enor latching disable bits inhibit those faults from triggering the error registers. However, if a disabled error is present and non-disabled error occurs, both errors will be recorded in the error registers.

Producing Errors

The Cyclops and Gambit gate arrays have specific registers which allow software to produce various errors on the Xbus.

The basic logic to produce an error consists of XOR gates on all the info, trid, func op, parity, control lines going out and all the same lines coming in. One input on the XOR is the data line the other input is the error line and the output is the resultant data. If the error line is high the data coming out will be in error. If the error line is low then the data coming out will be in the form of a loopback error. If the data is enored on the input and the output then the error will be in the form of a bus error since it will not be a loopback enor.

Table 5 shows all the error cases and the necessary settings from a hardware point of view to produce the error. 'Receive error' means the XOR gate on the receive side is inverting. 'Transmit error' means the XOR gate on the transmit side is inverting. The category describes if the error occurs only on a particular pattern match or if the error occurs just after the register is written. For many cases it would be possible to produce an enor with

different cases than those described below. The goal is to only produce the error in one mann

Table 5. Error Settings

CPU Board Error

Type of Error Category D-Side C-Side

CPU Board Faulty Input Circuit - CPU Immediate receive error no error Driving

CPU Board Faulty Input Circuit - I/O Data Match receive error no error

Board Driving

CPU Board Different Data C-Side and D- Immediate transmit error no error Side /receive error

CPU Board Faulty Output Circuit - Buffer Immediate transmit enor no enor to Pad Fault

CPU Board Open - CPU Board Driving Immediate no error receive error

CPU Board Open - I/O Board Driving Data Match receive error no error

CPU Board Short Immediate transmit error no error

Backplane Error

Type of Error Category D-Side C-Side

Backplane Open Etch

Backplane Short

IO Board Error

Type of Error Category D-Side C-Side

I/O Board Faulty Input Circuit Immediate receive error no error

I/O Board Output Circuit Fault - Buffer to Immediate transmit error no error Pad

I/O Single-side Access - ASIC Parity Immediate parity gen. no error Gen. Fault fault

I/O Single-side Access - ASIC PCI Data Immediate bad address no error

Path Fault

I O Non-single-side Access, Different C- Immediate transmit error no error D Data /receive error

I/O Board Open Immediate no error receive error

I/O Board Short Immediate transmit error no error

Transient Error

Transient Fault This causes a bus error, but when the error sequence is executed there will be no error. This fault will not produce a maint. int. from this board, but all the other boards will assert maint. int.

Bus Busy

Bus Busy This causes a bus busy.

Board Breaking Timing

Note in the figures Figures 14 - 17, Info O's shows the time the info bus is driven to zero assuming that the bus is not already tristated from this side, xcever disable shows the time the drivers from the broken board are disabled from the bus. Bus error is also shown when valid on the Xbus.

Board Breaking and Information Latching

There are several statos bits in the Broken Statos register on each board to determine the reason a board went broken. These bits are frozen after a broken condition occurs and will remain frozen until after a cold reset. All bits in the common register are straight forward with the exception of board specific set broken. Board specific logic may have several reasons to assert this signal and may also have a similar register for the reasons it has asserted this signal. If board_specific_broken is set in the common register, the board specific broken register will contain the real reason the board asserted the board specific set broken signal. If a board breaks for a reason other than board specific broken, the board specific broken bit will not be set in the broken statos register. However, board_specific_set_broken may have been asserted after the original broken condition. This would have caused the board specific broken statos register to latch and hold a reason it had set broken. It is important to understand that because the board specific broken bit is not set in the common statos register, this was not the reason the board went broken. The reason the board went broken is logged in the broken statos register and the board specific statos register may be ignored.

In the timing diagram of Figures 14 and 15, someone set broken is the logical OR of the following signals:

cold reset, warm_reset, slotjparity errorjset broken, control_or_3way_vote_sig_error, asic specific set broken, ecc dbl error in. In addition Gambit contains the following signals: break_pcib, xb_parity_gen_set_broken. The board will break if any broken signal

is asserted on any phase (i.e. any 48mhz edge), my set broken reg is the equivalent of otherside_set_broken_reg on the other ASIC. my_set_broken_reg is ORed with otherside set broken reg to create set broken sync.

The common broken statos register is frozen when latched broken statos is true. Note, All statos signals for the I/O register have the same timing as broken.

Note the bus error driven on the bus at the time the board went broken. This timing is important.

Board Breaking Timing on Info Loopback Error

As shown in Figure 16, if a loopback error occurs that will break the board, bus error will be asserted in POST2 and the board will break in ERRl. Both ASICs will know of the error because of there are four lines for each info bus that communicate loopback statos between the ASICs (cmp_eπ_in, cmp en out, drive en in, drive_eπ_ ut.

Board Breaking Timing on Heuristic or Arbitrary Broken.

As shown in Figure 17, boards that break due to a Heuristic or an Arbitrary broken will participate in the entire error sequence and then deassert board not broken in ERR2.

Xbus Physician Partitioning

This section gives a high level overview of the physical partitioning for the interface ASICs to the Xbus. It describes the components necessary to interface to the Xbus as well as a functional description of how the interface is intended to operate. The fault tolerant strategy is also described.

Info Bus Partitioning

As shown in figure 18, the C and D ASICs on the CPU board each drive half of the bits on the info bus when they are bus master. The C and D ASICs on the PCTB drive either half or all of the bits, depending on the type of transfer. During normal accesses, they drive half the bits as shown; during single-side accesses, they drive all of the bits, including the parity bit.

Control Signal Partitioning

The D-side ASICs drive the control buses, and both C and D-side ASICs check.

Xbus Voted Signal Partitioning

The 3-way voted signals in a Polo system have varying topologies depending on function. In general, each ASIC on a board drives half of the 3-way voted signals originating on a board, and the other side ASIC sits at the end of the net and checks what is driven.

Figure 19 shows the 3-way voted signals for board_not_broken_routing. Each transceiver section (shown as a NAND gate or inverter) of a 3 way voted triplet is in a separate package, to prevent a single failure from disturbing more than one bit of the triplet.

Signal Routing

The bidirectional info buses are routed point to point between boards. On each board, separate traces run from the connector to each ASIC; this allows the error protocol to determine if a fault is in the backplane or in the board. Series resistors are used to control signal quality, nd CMOS levels re used for increased noise margin and reduced susceptibility to cross-talk. Figure 20 shows the routing of info lines.

As shown in Figure 21, the three way voted signals are routed as separate point to point connections. The reset_x, y ,z signals are driven by the CPU boards, wire-ORed on the backplane, and received by all boards. The board not broken__ signal is driven by a single board and received by all boards. These signals are drive by normal 4mA drivers of the ASIC, and buffered by 26S10 trancseivers. The voted signals are terminated with a pull- up and isolation diode on both the receiver and driver side. This means that it is possible for there to be one or two pull ups on the tine. However, whenever the line is being used for activity signalling, there are exactly two terminators, one at either end of the line.

Functional Description

Figure 22 is a block diagram of the error check logic used by the ASIC's. The Error Checking and Registering logic performs two basic functions. It controls most functions or error checking and it registers incoming and outgoing Xbus signals. The error checking function performs the following tasks:

ECC generation and checking/correcting

Parity generation and checking

Loopback checking

Three-way voting

Controls error protocol (includes decisions on breaking board)

• Provides logic for error insertion

• Records error information

The registering logic performs the following tasks:

• registers incoming bus signals

• registers outgoing bus signals

• combines (logical 'or') neighbor, opposite, peer versions of registered ECC corrected control signals coming into the ASIC with the delayed outgoing corresponding signal to produce a combined control signal for internal ASIC module use

The Xbus can be broken into four classes of signals when discussing error checking:

Xbus Signals: these signals are protected by one parity bit, this detects an odd number of errors. During normal accesses the D-side gate array drives approximately half of the signals and the C-side drives the other half. Please refer to Section 11.1 on page 96 for more detail on the signals that are driven by D-side and C-side. The D-side checks the data it has driven (loopback drive check) and the D-side checks the data that the C-side had driven with the D-side would have driven (loopback compare check). During single sided accesses (PCTB only) the driving ASIC drives all of the bus signals. The driving ASIC can do a loopback drive check but the other ASIC cannot do a loopback compare check because it has no knowledge of the exact data being driven.

Control Bus Signals: these signals are protected by four bits of ECC, this detects a double bit error and can correct a single bit error. The D-side drives the signals and the C-side does the loopback compare check. The D-side does the loopback drive check.

Voted signals: these signals are triplicated and protected by three-way voting, this allow for one out of the three lines to be in error. The D-side drives the signals and the C-side does

the loopback compare check. The D-side does the loopback drive check. The one exception to D-side driving all signals is the board not broken signal. The C-side will drive the board not broken signal and the D-side will drive the enable for the board not broken signal .

Miscellaneous signals: The slot id signals are duplicated and they are in disagreement the board will break.

Figure 10 shows the Bus Error flow chart, summarizing the foregoing. A further understanding of the invention and of the construction and operation of the illustrated embodiment may be attained by references to the Appendices filed hereiwth.

Summary

Described above is a digital data processing system meeting the aforementioned goals. Those skilled in the art will appreciate that the illustrated embodiment is but one example of the invention and that other embodiments incorporating modifications thereto fall within the scope of the invention, of which we claim:

This document contains confidential and proprietary information of Stratus Computer, Inc., and any reproduction, disclosure, or use in whole or in part is expressly prohibited, except as may be specifically authorized by prior written agreement or permission of Stratus.

Polo Programming Guide

JED-00152

"A SAM is a logical concept"

Jeffrey Somers

Gregory Green

Michael Homberg

Conrad Clemson

Will Leavitt

Joe Lamb

26 December 1995

Revision: 3.2

Notice: Released Version

(c) Stratus Computer, Inc. 1995

Appendix I g- ø.

Polo Software Progran .ntng Guide Strata jmpany Confidential

1. Revision History 7

1.1 Open Issues 7

1.1.1 Polo UPS communications 7

2. Introduction 8

2.1 Applicable Documents 8

2.1.1 Stratus Applicable Documents 8

2.1.2 Other Applicable Documents 8

2.2 Terminology 8

3. Functional Overview 10

3.1 Polo Functional Overview 12

3.1.1 Logical Summary 12

3.1.2 Physical Summary 12

3.2 Jetta/FTIO Functional Overview 15

3.2.1 Logical Summary 15

3.2.2 Physical Summary 15

3.3 Jetta/HSC2 Functional Overview 16

3.3.1 Logical Summary 16

3.3.2 Physical Summary 16

4. Xbus/Golfbus to PCI Address Mapping 17

4.1 Addressing Overview 17

4.2 Address Re-mapping 18

4.3 Per-Space Ordering of I/O Accesses 18

4.4 Address Decode 18

4.4.1 Per-Slot I/O Space (Local, Paired, Non-Paired) 20

4.4.2 Global Broadcasts 21

4.4.3 Illegal Abnormal I/O Accesses 21

4.5 PCIB MIO/IOBus Compatible System Address Map 22

4.5.1 lOBus or Xbus to PCI Local Bus Requests 24

4.5.2 MIO Compatible Register Space 25

4.5.3 lOBus Compatible I/O Space Map 25

4.5.3.1 SAM I/O Address Space 26

4.5.4 lOBus Compatible Memory Space Map 28

5. Xbus/Golfbus PCI to Xbus Golfbus Address Mapping 29

5.1 Software View of Re-mapping 29

5.2 Polo Re-mapping 29

5.3 PCI Memory and I/O Address Mapping 31

5.3.1 PCI to Xbus/Golfbus I/O Requests 31

5.3.2 PCI to Xbus Memory Requests 32

5.4 Address Re-mapping 33

5.4.1 System Address Generation 33

5.4.1.1 Polo System Address Generation 34

5.4.12 HSC2 System Address Generation 35

5.4.2 IO Address Table Entry Invalidate 35

5.4.3 Address Error Checking 35

5.4.3.1 IO Device Address Checksum 36

5.4.32 Out of Bound Access Check 36

5.4.4 Control Bits 37

5.4.4.1 Valid Entry bit 37

5.4.4.2 Data Pre-read 37

29 December 1995 2

Polo Software P -gramming Guide .dtus Company Confidential

5.4.42.1 Coherency Restrictions 37

5.4.4.3 Lock Cycle 38

5.4.4.4 Swap 32 bit Endian 38

5.4.4.5 Incoherent Memory Access 38

5.5 Software Map Management 39

5.6 Byte Alignment 39

6. Interrupts 42

6.1 PCIB/HSC2 Interrupts 42

6.1.1 Introduction 42

6.1.2 Implementation goals 42

6.1.3 Hardware Implementation 42

6.1.4 Suggested Software Implementation 44

7. Boot 45

7.1 Boot Alternatives 45

7.1.1 Baby BIO Boot 45

7.1.2 Bio Firmware on Raid Controller Boot 45

7.1.3 CPU Prom Resident PCI Card Boot 45

7.1.4 Expansion ROM Resident PCI Card Boot 46

7.1.5 x86 Emulator PCI Card Boot 46

7.1.6 PCMCIA Flash PROM Based Boot 46

7.1.7 Conclusions 46

7.1.8 Open Boot 47

7.2 PCI Boot Process 47

7.2.1 Find and Configure the Boot SAM 47

7.2.2 Configure PCMCIA Bridge Chip 48

72.3 Configure PCMCIA Bus 49

72.4 Configure PCMCIA Flash Card 50

8. PCI Configuration 51

8.1 PCI Configuration Register Space 51

8.1.1 Gambit PCI Configuration Space 51

8.1.2 PCI Adapter Configuration Space 53

8.1.3 PCI Configuration space header description 54

82 Proposed configuration sequence 56

8.3 OS Based full system Configuration 56

8.4 PCI device detection 56

8.5 PCI Address formation 57

8.5.1 Configuration cycle generation 57

8.5.2 Special cycle generation 57

8.5.3 Polo/HSC2 Configuration Address Generation 58

8.6 Multi-function Adaptors 59

8.7 SAM and PCI Adaptor Configuration 59

8.7.1 Base Address Registers 59

8.8 PCI Address Space 60

8.9 PCI Bus Target and Master Aborts 60

8.9.1 PCI Bus Fault Detection and Tolerance 61

8.10 On Line Adds 61

8.11 On Line Failures 61

8.12 Board removal 61

8.13 Special Requirements 61

29 December 1995

7/09781

Polo Software Programming Guide Stratus ι jmpany Confidential

8.13.1 Lock requirements 62

9. PCIB Maintenance & Diagnostics 63

9.1 Overview 63

92 ID Proms 63

92.1 Fault Logging 63

92.1.1 PCIB 63

9.2.12 HSC2 64

922 I/O Power Supply Faults 65

9.3 Insertion & Removals 65

9.3.1 PCIB devices 65

9.3.2 Polo PCI devices 66

9.4 Reset Overview 66

9.4.1 Polo Resets 66

9.4.1.1 CPU Reset 66

9.4.12 PCIB 67

9.4.1.3 PCI cards 69

9.4.2 HSC2 Resets 70

9.4.2.1 HSC2 70

9.422 PCI cards 71

9.5 Determining the source of a Maintenance Interrupt 72

9.6 Faults & Errors 72

9.6.1 PCI card faults 72

9.6.2 PCI bus faults 73

9.62.1 PERR# 73

9.622 SERR# 73

9.62.3 Polo(Gambit)/HSC2(Mirage) Error logic 73

9.6.3 Disk bus faults 76

9.6.3.1 LED Initialization 76

9.6.32 Disk LED Control 76

9.6.3.3 Disk Insertion and Removal 76

9.6.4 SCSI bus faults 77

9.6.5 PCIB/HSC2 faults 77

9.6.6 Xbus/Golfbus faults 77

9.7 Diagnostics 77

9.7.1 PCI Card 77

9.72 PCIB 77

10. ReCC Functionality on Polo 78

10.1 Remote Power control 78

102 RS-485 DUS 78

10.3 Powerfail 78

10.4 Clockcard, backpanel power supply failures 78

10.5 I/O power supply failures 78

10.6 NVRAM 78

11. Board States 79

11.1 PCIB Board States 79

112 HSC2 Board States 79

11.3 SAM Board States 80

12. Register Definitions 82

12.1 XBus/Golfbus Bus Interface Registers 82

29 December 1995 . 4

Polo Software Guide Status Company Confidential

122 Reading Registers with Different C/D Information (Polo Only) 82

12.3 Common Xbus Golfbus Register Definitions (Polo Only) 83

12.3.1 ID Prom lnstr 85

12.3.2 ID PROM Address Data 86

12.3.3 Board Reset 88

12.3.4 Bus Interface State 90

12.3.5 Board Sync Register 93

12.3.6 LED Control 95

12.3.7 Slot ID 96

12.3.8 Read Ping Interval 96

12.3.9 General Purpose Communications f :0] 97

12.3.10 Memory Size/Location 97

12.3.11 Test Control 97

12.3.12 Bus Interface Fault Reporting 98

12.3.13 Common Broken Status 100

12.3.14 ASIC Specific Broken Status 101

12.3.15 Bus Info Error Status 102

12.3.16 Misc. Error Status 103

12.3.17 Control Bus Error Status 104

12.3.18 Bus Error Byte Status 105

12.3.19 Voter Error Transceiver Status 106

12.3.20 Bus ASIC Chip Revision 107

12.321 Performance Counter 107

12.3.22 Performance Counter Trig[1:0] 108

12.323 Performance Counter Mask[1:0] 108

12.324 Performance Counter Control 109

12.325 Fault Bit[1:0] 109

12.326 Data Match 110

12.3.27 Error Control 110

12.4 CPU Specific Xbus Register Descriptions 112

12.4.1 ASIC Specific Configuration 112

12.42 I/O Address Map Error 1 112

12.4.3 I/O Address Map Error 0 113

12.4.4 Time Of Day 113

12.4.5 Quicktime 114

12.4.6 Master Jiffy Counter 114

12.4.7 Jiffy Control 114

12.5 PCIB/HSC2 Specific Xbus Register Descriptions 115

12.5.1 Gambit Maintenance Attention Request [31 :0J 115

12.52 lOBus Status 116

12.6 SAM Interface Register Descriptions 117

12.6.1 SAM Interface Registers - Page 3 119

12.6.1.1 Disk LED Control 119

12.6.12 Disk Status 119

12.6.1.3 Power Supply LED 121

12.6.1.4 Power Supply Status(Vanguard powe supply) 121

12.6.1.5 Fan Speed Control 121

12.6.2 SAM Interface Registers Page 2 122

12.62.1 Address Table Command and IOVA Register 122

12.622 Address Table Data 2 Register 122

12.62.3 Address Table Data 1 Register 123

12.62.4 Address Table Data 0 Register 123

12.6.3 SAM Interface Registers- Page 1 124

12.6.3.1 PCI config_addr 124

29 December 1995

Polo Software Programming Guide Stratus _ ompany Confidential

12.6.32 PCI config_data 125

12.6.3.3 Test Control 125

12.6.3.4 PCI Error 126

12.6.3.5 PCI IOVA Error 127

12.6.3.6 SAM Status 127

12.6.3.7 Arbitration Freeze Count Max. Value 128

12.6.3.8 Host Request FIFO Timeout Value Register 128

12.6.4 SAM Interface Registers - Page 0 129

12.6.4.1 Board Reset 129

12.6.42 LED Control 130

12.6.4.3 SAM Host Interrupt Bit 130

12.6.4.4 SAM Interrupt Mask 131

12.6.4.5 SAM Interrupt Source 131

12.6.4.6 SAM Host Interrupt Address Pointer 131

12.6.4.7 SAM Host Interrupt Table Pointer #2 131

12.6.4.8 SAM Host Interrupt Table Pointer #1 131

12.6.4.9 SAM Configuration 132

12.6.4.10 PCI IO Space Offset 132

12.6.4.11 PCI Memory Offset 132

12.6.4.12 Bus Interface State 132

29 December 1995

Polo Software F. Guide atus Company Confidential

1. Revision History

Rev 0.0: April 26, 1994 Created.

Rev 1.0: Aug 8, 1994

• Many updates for software

Rev 1.1: Aug 10,1994

Updates from internal review for sections up to interrupts.

Rev 2.0:

• Release Version

Rev 22: March 21, 1995 FTIO updates

Rev 3.0: July 19, 1995

Removed obsolete FTIO information

Rev 3.1 : September 1 1995

• HSC2 Updates

Rev 32: SDecember26, 1995

• Internal Review HSC2

1.1 Open Issues

Open Polo related issues are also tracked in problem under pci_common. This list reflects open issues that will not be closed for the current release of the specification along with a justification for the issues to remain open.

Open HSC2 related issues are tracked in problem under hsc2_issues. This list tracks unresolved issues during the development cycle of the HSC2 project.

1.1.1 Polo UPS communications

Polo will specify, support and qualify an external UPS solution. The actual vendor selection for the UPS has not occurred yet. Only vendors with an RS-232 interconnect are being considered. When the vendor has been selected, the section on UPS support will be updated to provide the communications specifications.

26 December 1995

Polo Software Programming Guide Stratus v_ j pany Confidential

2. Introduction

This speαfication is the Polo Programming Guide tor the Polo PCIB based system. It provides both the functional model for the hardware and the software detailed design specifications. It is intended to be the pnmary working model for the programming guide, and it specifies the software visible hardware implementation.

2.1 Applicable Documents

2.1.1 Stratus Applicable Documents

• Cougar Specification, JED-0005

• Xbus Functional Specification, JED-0155

• Cyclops ASIC Functional Speαfication, JED-0159

• Gambit ASIC Functional Speαfication, JED-0160

• Polo Product Specification, JED-0151

• Polo Backplane Specification, JED-0158

• Xbus Functional Specification, JED-0155

• Mirage ASIC Functional Specification, JED-01 f>

2.1.2 Other Applicable Documents

• PCI Local Bus Specification Rθv-2.0, April 30, 1993 PCI Special Interest Group

• PCI System Design Guide Rev-1.0, September 8, 1993 PCI Special Interest Group

• PCMCIA 2.1/JEIDA 4.1 Specification

2.2 Terminology

• Cyclops - The Polo Ibus to Xbus ASIC. This ASIC contains many of the features of the Gofer ASIC.

• Gambit- The PCIB based PCI Xbus interface ASIC.

• Mirage - The HSC2 based PCI/Gbus interface ASIC.

• QBI - Golfbus(Xbus) Interface ASIC. This is the Gofer in Jetta systems and the Cyclops in Polo systems.

• Rash ram - Flash RAM in this speαfication refers to a PCMCIA flash ram card. The flash ram card is used tor boot functionality.

• FTIO - FTIO is the name for the overall I/O subsystem that encompasses the MIO board, the fault tolerant I/O bus, and the SAM and PCI cards.

• Gofer- The Jetta based GBI ASIC.

• HSC2 - The Jetta Golfbus/PCI interface board

• IOVA - I/O virtual address.

• lOBus - The fault-tolerant bus between the MIO board and the SAM modules.

• MIO - The Jetta Golfbus IOBus interface board.

• OpenBoot - This is a proposed standard interface for booting any OS from a PCt card. It is independent of the operating system, the system hardware and the microprocessor This standard ts stilt under negotiation.

26 December 1995 8

Polo Software Pi gramming Guide _^ -atus Company Confidential

Polo - The new low end system. The Polo system is fully described in the Polo Product Specification.

PCI - Peripheral Component Interface; PC Bus under development by Intel and others as a standard for PCs. The expectation is that peripheral device controller chip sets will be available which can reside directly on the PCI bus PCIB - The Polo equivalent of a SAM module.

PCMCIA - PC Memory Card International Association. PCMCIA is an industry standard bus which is used for boot functionality on Polo and FTIO.

PMI - Processor/Memory Interface ASIC. This is the Cougar in Jetta and Polo systems SAM - Stratus Adapter Module - A Jetta-based board which provides the fault-tolerant multi¬ drop interface to the lOBus. SAM is a generic term for lOBus adapter (not just PCI board adapters). This specification deals only with the SAM/PCI version of the SAM adapter. TRID - Transaction ID - This binary number provides an identifier for Golfbus and Xbus operations.

26 December 1995

Polo Software Programming Guide Stratus Company Confidential

3. Functional Overview

Figure 1 below shows a block diagram of a one SAM bucket system that both the FTIO, HSC2, and Polo systems present to software. This diagram is meant to show only the programming model and not the actual implementation.

For a Polo system, software should view the system as a 4 slot Golfbus based system. There are Mercury CPUs in slot 0 and 1 and MIO IO boards in slots 2 and 3. The MIOs in a Polo system will never be in a duplexed state.The MIO can connect to one PCI bucket with up to 16 SAMs in it.

For an FTIO system, there can be CPUs in slots 0-3. MIOs can reside in slots 2-11. The system can support multiple MIOs. Each MIO can support 1 to 64 SAM cards residing in 1 to 4 SAM buckets.

For an Jetta based HSC2 system, there can be CPUs in slots 0-3 and MIOs can reside in slots 4- 11. The MIOs in a Jetta system will never be in a duplexed state. The MIO can connect to 1 PCI bucket with up to 4 SAMs in it.

Throughout the document, many of these basic blocks are referred to in both Polo and FTIO descriptions. Although mentioned in the terminology section, several of these components are extremely important in understanding of how both systems work and present a compatible view to software. For this reason, these blocks are described below. In some cases, these units are real hardware, in others these blocks are pure abstractions, representative of potential hardware, or merely first instantiations of potential hardware.

29 December 1995 .. 10

Polo Software Programming Guide wa-atus Company Confidential

Figure 1. Logical High Level Diagram of the Polo and FTIO System

Golfbus Xbus

Mercury CPU board: the mercury CPU board is the first CPU board in the Jetta product family. The Mercury CPU board is expected to be the first CPU board in the FTIO product. While a modification of this CPU is used in Polo, it is identical from a software perspective to the Mercury CPU. When the term CPU or Mercury CPU is used in this document it refers to either the Polo or FTIO CPU and any follow on CPU product

FLASH Card: PCMCIA card containing Flash (non-volatile) memory. Information can be organized as an ATA disk or as memory.

FTIO: Fault-Tolerant IO subsystem consisting of an MIO pair, lOBus, Repeater pair, SAM and SAM backpanel. The subsystem offers fault-tolerant connectivity to an open bus standard, namely PCI.

Golfbus Xbus System Bus: FTIO and Polo use different system buses. Golfbus is the Jetta system bus. Xbus is the Polo system bus. Except for M&D these buses are identical from a software perspective. The M&D differences are derived form the different approach the buses must take to bus errors. Golfbus, Xbus, and system bus are used interchangeably in this document to refer to the system bus interconnect lOBus: IO Bus is a Fault Tolerant IO bus unique to the FTIO implementation of the system. This bus is transparent to software except for M&D handling of error conditions.

29 December 1995 11

Polo Software Programming Guide Strata company Confidential

MIO: MIO is the FTIO interface board which connects the system bus to the lOBus. In the FTIO subsystem this is a single board duplexable unit In the Polo system, the MIO is an abstraction. Although not implemented as a unique board, its software visible aspects are implemented in different components of the system.

PCI Bus: the PCI bus is an Intel designed peripheral bus which is used as the industry standard IO bus tor both FTIO and Polo.

PCI Card: A PCI card is a 3rd party card which interfaces to the PCI bus and an I/O specific interface. A PCI card is the I/O adaptor for both systems.

PCIB: A PCIB (PCI Bridge) is unique to Polo. The PCIB performs the functions of the FTIO MIO, lOBus, and SAM. It is invisible to software, except from an M&D standpoint

PCMCIA: Personal Computer Memory Card Interface Association. Interface Standard used typically for lap tops and notebook pcs for memory and i/o devices. The cards are 54mm wide by 85.6 mm long (credit card size). The cards are available in 3 heights, 3.3mm, 5.0mm and 10.5mm.

SAM: The SAM is the interface and control unit for the PCI cards. In the FTIO system, the SAM is a hot pluggable card which performs isolation and fault tolerance for the PCI cards. In the Polo system, the SAM is an abstraction tor the logic and software visible aspects of PCI card control. In this document the term SAM applies to the software visible features associated with control and fault tolerance with respect to the PCI bus.

SAM Repeater: The SAM repeater is a unique SAM which interfaces to a PCMCIA bus. In the FTIO this card also performs some low level electrical and M&D functions.

3.1 Polo Functional Overview

The Polo Product is a Jetta Mercury and FTIO only based system. From a high level, some of the key features of the CPU board are:

• PA7100 Based Microprocessor, 72 or 96 Mhz, 256KB or 1 data and instruction cache.

• 1 or 2 microprocessors per board

• 128MB.256MB, 512MB or 1 GB of DRAM memory.

This implementation of the MIO subsystem has the following key features:

• Standard I/O Bus support (PCI)

• 1 to 16 PCI cards supported.

3.1.1 Logical Summary

Figure 1 above is a logical diagram of the overall system. As pointed out many of the specific features, SAM, lOBus, and MIO, are abstracted in the Polo system.

3.1.2 Physical Summary

While the logical view of the system is that of a standard Golfbus machine, the physical view is very different figure 2 below illustrates an overview ot the Polo system.

There are two major logical components that will be designed by Stratus in this figure. The first major component is the mother board consisting of the Mercury core (snoop subsystem, cougar, 1

29 December 1995 12

Polo Software > lOgramming Guide όtratus Company Confidential

or 2 Lisbon memory cards, and on or 2 logical PA modules) and ReCC. The new ASIC on the motherboard, the Cyclops ASIC, contains all software visible aspects of the Gofer and MIO Golfbus ASICs. From a software perspective, the Golfbus is inside Cyclops. As explicitly defined in later sections, all internal registers are either implemented or will return intelligent responses to software operations.

The second major component is the PCI interface board. This board contains two instances of the CPU-PCI interface ASICs, the Gambit ASIC. Each Gambit acts as the interface between the CPU and one PCI bus. Each Gambit emulates the functionality of 4 SAM cards (one for each PCI slot).

The interconnect between the CPU boards and the PCI interface boards is a set of cross-couple buses. These buses are fault detecting, but not correcting. The buses look like a hybrid Golfbus and are named the Xbus. A complete description of the bus interface can be found in the Xbus Functional Specification. Memory updates, Regurgitated infos, and other CPU to CPU traffic is run across the Xbus.

In Polo, the MIOs will always appear to be simplexed. All accesses can be performed to either slot 2 or slot 3 address space, but must be performed as non-paired accesses. Paired accesses will be responded to as TBD. Just don't do it...

29 December 1995 13

0 781

Polo Software Programming Guide Stratus company Confidential

Figure 2. Polo System Overview

Board 0 Board 1

Power Supply Power Supply

Non-FT power tor Non-FT power tor PCI slots and disk PCI slots and disk

29 December 1995 14

Polo Software Programming Guide ^..atus Company Confidential

3.2 Jetta FTIO Functional Overview

The FTIO project was cancelled mid-project. Although it is no longer a planned project, there was a tight compatibility goal between polo and FTIO. The remnants of this remain in the design. For this reason, the following sub-sections are included in the specification to provide background to the reader.

This FTIO/MIO/SAM implementation has the following key features:

• Standard I/O bus support (PCI and PCMCIA).

• Up to 64 SAMs (Total of Repeaters plus SAMs)

3.2.1 Logical Summary

The logical view of the FTIO sub-system is consistent with Figure 1.

3.2.2 Physical Summary

FTIO is a standard Jetta Golfbus interface Sub-system. There are four major logical components that will be designed by Stratus in this figure. The first major component is the MIO primarily consisting of the IF ASIC. The IF ASIC provides a bridging function between the Golfbus and the lOBus. There is no processor or memory resident on MIO.

The second major component is the Repeater. The Repeater boards receive the point-to-point differential ECL lOBuses and converts the levels to BTL for use in the multi-drop SAM backplane. The Repeaters provide power for the SAM backplane termination and also re-broadcast the differential ECL lOBuses to allow daisy-chaining up to 4 SAM chassis. (Each SAM chassis contains 2 Repeaters and up to 14 SAMs) Each Repeater consists of ECL and BTL bus transceivers and an IAM ASIC. The IAM ASIC provides the host-bridging function from lOBus to PCI Local bus. The IAM also provides Repeater specific support when used in the Repeater mode. The Repeater uses a 3-way voted clock generation circuit. The Repeaters also support a type II or III PCMCIA interface.

The third major component is the SAM. Each SAM consists of BTL bus transceivers and an IAM ASIC. The IAM ASIC provides the host-bridging function from JOBus to PCI Local bus. Each SAM supports a single 32-bit PCI slot The IAM provides system memory protection for bus mastering PCI adapters and also SAM specific support when used in the SAM mode (as opposed to repeater mode). The SAM uses a 3-way voted clock generation circuit

The fourth component is the SAM Backplane/SAM clock board combination. The SAM backplane is a 16 slot backpanel which supports 2 Repeater and 14 SAMs. The backpanel will conduct the bulk power to each SAM and Repeater, eliminating the need for bus bars. The backpanel will also provide the cable attaches for the lOBus cables, bulk power connections, CDC connections and a JTAG port for scanning each slot individually. The backplane will also have connectors for mounting the SAM clock board. The SAM clock board will provide point-to-point triplicated clocks to each SAM and Repeater board. The first iteration of the clock board will be a high-reliability model. A follow-on project can replace the high-reliability clock with a fault-tolerant clock. The high-reliability clock cannot be replaced on-line.

29 December 1995 15

T-

T/US97/09781

Polo Software Programming Guide Stratus Company Confidential

3.3 Jetta/HSC2 Functional Overview

The HSC2 project started during first release of Polo hardware. One of the goals of HSC2 is software compatiblity with Polo tor use in Jetta-12 and Jetta-6 configurations. Sections of this document are being updated to highlight differences and give the reader insight

The Jetta HSC2 implementation has the following key features:

• Standard I O bus support (PCI and PCMCIA).

• 1 to 4 PCI cards supported per Jetta slot.

3.3.1 Logical Summary

Figure 1. is a logical diagram of the overall system. As pointed out, many of the specific features; SAM, lOBus, and MIO are abstracted in the Jetta/HSC2 system.

3.3.2 Physical Summary

The logical and physical view of the system is that of a standard Golfbus machine. Figure 3 below illustrates an overview of the Jetta/HSC2 system.

Figure 3. HSC2 System Overview

Golfbus

The major component that will be designed by Stratus is the HSC2 Jetta I/O board. The HSC2 is a simplexed board so all accesses must be performed as non-paired accesses.

The new ASIC on the HSC2 board, the Mirage ASIC, contains all the software visible aspects of the MIO Golfbus ASIC and also emulates the functionality of 4 SAM cards (one for each PCI slot).

mber 1995 16

Polo Software F-.ugramming Guide w fatus Company Confidential

4. Xbus/Golfbus to PCI Address Mapping

This section outlines many aspects of addressing on the Polo PCIB and Jetta HSC2 systems. The overall high level address map is presented and PCIB/HSC2 specific aspects are outlined. The address translation map is presented and its outline and management are discussed.

4.1 Addressing Overview

The Polo system address map is designed to preserve maximum compatibility with the Golfbus address map. As such, it preserves the support for 15 logical slots, etc. even though Polo is a fixed configuration of two CPU boards and two I/O boards.

The PA7100's address map is broken down into two segments, main memory and I/O space. All of main memory is cacheable and all of I/O space is non-cacheable. There are no limitations as to what virtual address is mapped into what physical I/O space.

Figure 4. PA7100 Physical Address Map

0x00000000

OxFOOOOOOO

Main Memory

Address

Space

fiO ^"

Address Space

OxFFFFFFFF

OxFFFFFFFF

By mapping I/O space into the virtual address space and using the same page translation mechanism as that employed for memory, the PA7100 architecture allows the flexibility for non- privileged users to be given access to some regions of I/O space without compromising system security. All duplexed boards' I/O register definitions can be accessed through local, paired, non- paired, and global space and all simplexed boards' I/O register definitions can be accessed through local, non-paired, and global space. MIO mirrors it's Golfbus registers so that they can be accessed in "freeze space" to allow a CPU controlling the duplexing process to access MIO's registers while MIO is in freeze state.

• Local Space - Resource is local (same-board) to the requester and therefore does not require an Xbus Golfbus cycle. Registers such as CPU interval Timers, Interrupt & Mask Registers are good examples of frequently accessed registers that are optimally handled locally. When a board is off-line it can only access and test I/O on-board through local space.

29 December 1995 17

Polo Software Programming Guide Stratus Company Confidential

• Non-Paired Space - I/O space is accessed explicitly by slot number. Note that boards that are duplexed are required to go through the motions on non-paired I/O but only the addressed slot will perform the wnte for write operations or dnve data onto the XbusGolfbus for read operations.

Paired Space - I/O space is accessed explicitly by slot number; however the least significant bit, which differentiates the even/odd slot of a duplexed pair, is ignored. There are registers in I/O space that are not allowed to be read through paired space such as a per-board broken status register.

• Global Broadcast Space - Global I/O space is defined as write-only space (software needs to enforce this). It's pnmary purpose is to provide a means of broadcasting synchronous commands to multiple boards in the backplane simultaneously.

• Freeze Space (Paired and Unpaired) - This "MIO only" space resides at offset 7FExxxx and allows a CPU that is handling the duplexing of the MIOs to access the Golfbus I/O registers while MIO is in freeze state. Normal paired and unpaired accesses (offset 7FFxxxx) get busied.

4.2 Address Re-mapping

Polo/HSC2 use address re-mapping to protect the system from errors caused by simplexed I/O devices. Note that the I/O Virtual Address or IOVA format used by Polo & HSC2 systems differs from the IOVA format used on BIO systems.

In HSC2 systems the IOVA is translated into a system address inside the Mirage ASIC on the HSC2 board. The Map RAM is also internal to the Mirage ASIC which is self-checking.

In Polo, all system accesses trom the I/O ASICs are performed using the IOVA (IO virtual address); these are mapped to the final Xbus style physical virtual index addresses (CPU addresses) by a map RAM in the Cyclops ASIC. Both IOVA and CPU addresses are passed on the backplane, with a bit in the trid field distinguishing between them

4.3 Per-Space Ordering of I/O Accesses

IO accesses requiring the Xbus/Golfbus (accesses to paired, non-paired, or global space) are ordered relative to each other for a given processor. I/O accesses to local space are only ordered relative to other local I/O accesses from the same processor. In particular, if a write to a register in global space is followed by a read of that register in local space, it is possible for the read to occur before the wnte takes place. Venfication of global writes should be performed by non-paired reads.

There is no ordenng guaranteed between I/O space accesses and memory space accesses. A mechanism for making I/O accesses sequential to memory accesses is board specific and descnbed in the relevant board specifications.

4.4 Address Decode

The Golfbus I/O address map used by Polo and HSC2 systems is subdivided into 16 segments of iβMbytes each. The first 15 segments correspond to Slots 0-14 in the backplane; slot 15 is reserved to map local and global I/O Addresses. A Polo system looks to the programmer like a four slot Golfbus backplane. Note that the slot identifier field of the I/O address is inverted to allow local address space to begin at absolute address OxFOOOOOOO which corresponds to the boot address issued by the PA7100 upon being released from reset.

199 18

Polo Software P. ^ramming Guide c-atus Company Confidential

Where: ss"§s is the slot # inverted:

1111 -> Slot#0 0000 -> Slot #15

Each slot with the exception of slot 15 is subdivided into paired and non-paired space which effectively halves their logical address space from iδMbytes to SMbytes. All registers on duplexable boards are addressable through either paired or non-paired space. Accesses to paired space on a non-duplexable board are ignored.

29 December 1995 19

9781

Polo Software Programming Guide Stratub company Confidential

Figure 6. Physical I/O Slot Decode

4.4.1 Per-Slot I/O Space (Local, Paired, Non-Paired)

In general the view of XbusGolfbus I/O space is identical whether it is addressed through local, non-paired, or paired space. All board I/O registers will reside in the last 16 pages of I/O space and each board in the Poio/HSC2 architecture will have a common set of registers that reside on the last physical page of the per-slot I/O segment to simplify the M&D (Maintenance & Diagnostic) interface. Additionally the last 1 Mbyte segment of a board's address space can be touched via global broadcast write operations, which are described in greater detail in the next section.

29 December 1995 20

Polo Software ι .ogramming Guide jtratus Company Confidential

Figure 7. I/O Space Map

Global Broadcast Non-Paired Space Paired Space

Global Broadcast Non-Paired Space Paired Space

? = inverted board slot #

4.4.2 Global Broadcasts

Global broadcasts are write-only and are visible to all boards through slot 15 as illustrated in figure 6. Global broadcasts can be issued to all boards simultaneously in the backplane or to classes of boards such as all CPU/MEM slots. Global broadcast operations must be architected in such a manner that all boards that receive them perform the operation synchronously relative to one another. Figure 8 below indicates the decoding for global broadcasts.

Figure 8. Xbus/Golfbus Global I/O decode

31 2827 242322 20 19

CCC = Command Type

000 = Broadcast to all Boards

001 = Broadcast to CPU/Memory Boards 010-111 = Reserved for Other Board Classifications

4.4.3 Illegal/Abnormal I/O Accesses

Table 1 summarizes the behavior of various types of illegal/abnormal I/O accesses. Note that as described in section 12.1 on page 82, though all common registers are 32 bits, they are spaced at 64 bit (8 byte) intervals to aid future board designs which may make use of 64-bit internal buses.

29 December 1995 21 9

/US97/09781

Polo Software Programming Guide Stratum „ompany Confidential

Also note that the common logic provides an indication to board specific logic that a read access has timed out/no-ACKed, or that a write access has failed (i.e. a simplex source broke after successfully driving info but before driving data). Different boards handle these conditions in different ways. Mercury boards for example:

• return O's and signal LPMC's (HP-PA low priority machine checks) for timed-out/NACKed reads

• do nothing tor failed writes other than to write O's or nothing (per the table below) .

Table 1. Illegal/Abnormal I/O Registers

4.5 PCIB MIO/IOBus Compatible System Address Map

All of the MIO compatible address space can be addressed directly through the Xbus/Golfbus I/O space. Each I/O boards' address space can be accessed directly via set size windows, where the base address of the window can be programmed individually on each board.

The Polo version of the MIO compatible map is a subset of the map presented in the following sections. As previously stated the Polo appears to software as a system with simplexed MIOs in slots 2 and 3.

29 December 1995 22

Polo Software Programming Guide buatus Company Confidential

The Polo PCIB and Jetta HSC2 will only operate in simplexed mode. All lOBus PCI slots are decoded according to the address map defined bellow.

Figure 9 shows the MIO/IOBus compatible address map in context of the Xbus address space.

Figure 9. MIO/IOBus Compatible Address Map in Xbus/Golfbus Space

16MB of Slot-? I/O Space (figure 6)

The breakdown of the Xbus/Golfbus address bits is illustrated in table 2. Table 2. Xbus/Golfbus address decoding

MIO Compatible Register Space:

• The register space includes all registers of the I/O board, including the standard Xbus/Golfbus I/O registers, the ASIC registers, and the ID PROM space. Many register locations will be commonly defined for all I/O boards. See section 4.5.2 tor details.

29 December 1995 23

%

Polo Software Programming Guide Stratus company Confidential

lOBus Compatible I/O Space:

• This space includes registers and memory on the PCIB/HSC2; a portion of this address space will be turned into I/O cycles on the PCI bus. Only a 32 KB window of the PCIB/HSC2 I/O space can be accessed from the Xbus Golfbus at a time. The base offset of the window can be changed on the Mirage or Gambit. See section 4.5.3. lOBus Compatible Memory Space:

• Access to the memory space portion of the lOBus compatible address space will be turned into memory cycles on the PCI bus. Only a 64 KB window of the PCIB/HSC2 memory space can be accessed from the Xbus/Golfbus at a time. The base offset of the window can be changed on the Mirage or Gambit. See section 4.5.4.

4.5.1 lOBus or Xbus to PCI Local Bus Requests

Base address registers have to be set up in either the Mirage or Gambit ASIC and in the PCI configuration space of the PCI adapter card in order to allow I/O or memory (non-configuration) accesses to the PCI adapter across the PCI local bus from the Goflbus/Xbus. One set of registers (and hence one I/O and memory window) exists for every PCI slot. Figure 10 illustrates a high- level view of this mapping process.

Figure 10. SAM address window scheme

Direct memory accesses to the PCI adapter must be made through the 64KB SAM memory space window which can be shifted through the entire 32-bit adapter memory address space. I O accesses to a PCI adapter likewise must be done through the 32 KB SAM I/O space window. The adapter card has to be programmed to respond to a specific memory or I/O area where it will be addressed by the system. Each adapter type may have different memory or I/O range requirements. These requirements can be determined from the base address registers In the adapter's PCI configuration space. The PCI Memory Offset register of the Mirage or Gambit ASIC has to be set to point to the bottom of the selected memory range of the PCI adapter. The memory can start anywhere in the 4GB address range on a 64KB boundary. Also, the PCI IO Space Offset register of the Mirage or Gambit ASIC has to be set to point to the bottom of a 32KB window within the I/O range of the PCI adapter. See the specific PCI adapter card's specification and section 4.5, PCIB MIO/IOBus Compatible System Address Map, for more on this. Thus the procedure for accessing either memory or I/O space on an adapter card is:

29 December 1995 24

Polo Software Programming Guide stratus Company Confidential

initialize PCI Memory or I/O Space Offset register in the bndge (Mirage or Gambit) to the bottom of a 64KB (memory) or 32KB (I/O) window.

• initialize PCI Memory or I/O Space Offset register in the PCI adapter.

• access memory or I/O window through Golfbus/Xbus I/O space (section 4.5). Any access to the PCI address windows is interpreted as legitimate access using whatever address happens to be in the Offset Registers.

At a minimum the software should guarantee that the PCI slot memory space windows and the host bridge memory space do not overlap for any group of four PCI slots assigned to a single PCIB card. Figure 11 shows an example for a group of four PCI adapters. Alt four PCI adapters reside on the same PCI bus so care has to be taken in laying out the address windows.

Figure 11. PCI Bus Memory Space Map

32-bit PCI System Memory Address Map

LWWW ^l = NOT MAPPED

4.5.2 MIO Compatible Register Space

OxF%6F00O0 -0xF%6FFFFF: Xbus/Golfbus common and MIO specific compatible registers, including access to the ID PROM. For a detailed register address map and description see section 12.

The ^* %' is equal to the inverted Xbus/Golfbus slot number. The addresses listed are for non-paired space, for paired space addresses change bit 23 from a zero to a one.

4.5.3 lOBus Compatible I/O Space Map

The following lOBus Compatible I/O Device address map allows for 64 addressable I/O slots per lOBus. The address space is divided to I/O Space and Memory Space.

The Polo system only supports slots 0 to 15. The other 48 slots behave as un-populated slots in a

29 December 1995 25 "

T/US97/09781

Polo Software Programming Guide Stratus _^ mpany Confidential

Polo system.

The HSC2 system only supports slots 0 to 3. The other 60 slots behave as un-populated slots in a HSC2 system.

Figure 12. lOBus Compatible I/O Space

GoHBus address lOBus Address Space

ot #

3

Table 3 illustrates the Xbus Golfbus address formation for the lOBus compatible I/O space shown in the figure above. The slot field in the address is the inverted Xbus slot number.

Note that SAM/PCI slots 16 to 63 do not exist and the MIO compatible space exists in Xbus slots 2 and 3. Furthermore the Polo PCIB does not support duplexing so it will not respond to paired space accesses.

Note that SAM PCI slots 4 to 63 do not exist and the MIO compatible space exists in HSC2 Furthermore the Jetta HSC2 does not support duplexing so it will not respond to paired space accesses.

Table 3. lOBus compatible I/O space slot and offset bit positions

a. paired/ non paired bit

4.5.3.1 SAM I/O Address Space

SAM I/O space is divided up into two spaces (see figure 3).

29 December 1995 26

Polo Software Programming Guide S atus Company Confidential

PCI I/O Space: The PCI Local bus can support a full 4GB of I/O space, but the historical maximum is 64KB due to x86 limitations. Access to the 64KB PCI I/O Space ts made through a 16 KB window defmed as SAM PCI/IO Space. Accesses to the SAM PCI/IO Space are passed along to the PCI bus with the I/O Read or I/O Write function codes. The position of this 16 KB window can be changed on the 16 KB boundary via the PCI I/O Space Offset Register. The base I/O address of the PCI adapter must align with the PCI I/O Space Offset Register of the Mirage or Gambit ASIC. A read that maps to the PCI I/O space of the SAM and generates a PCI cycle that does not map to the I/O space of an adapter will cause a master abort and return zeros. The PCI bus read could also map to the I/O space of the next adapter if the I/O spaces of the adapters are contiguous.

When a sub word access is made to PCI IO space the byte enables are passed through from the read. Since the address from the CPU will have 2Tj00 in the address bits [1 :0] these two address bits need to be reconstructed from the byte enables as shown in table 4.

SAM compatible Register Space: Operations to this 16 KB space provide access to SAM compatible ASIC internal registers. Reads to registers that are not implemented in this space returns zeros.

Figure 13. SAM I/O Space Map

SAM

The following table illustrates the formation of system bus addresses within the PCIB/SAM I/O address space.

Table 5. SAM Compatible Register System Address Formation

29 December 1995 27 s

Polo Software Programming Guide Stratus company Confidential

a. paired space -> - 1; unpaired space -> - 0 b. 1 - SAM compatible ASIC register space; 0 - PCI I O space

4.5.4 lOBus Compatible Memory Space Map

Once again the Polo system only supports 16 slots and the other 48 spaces behave as un¬ populated slots in a Polo system.

Figure 14. lOBus Compatible Memory Space

Xbus address lOBus Address Space

#

Table 6 illustrates the Xbus/Golfbus address formation for the lOBus compatible memory space shown in the figure above. The slot field in the address is the inverted Xbus slot number.

Note that in Polo systems, SAM/PCI slots 16 to 63 do not exist and the MIO compatible space exists in Xbus slots 2 and 3. Furthermore the Polo PCIB does not support duplexing so it will not respond to paired space accesses either.

Note that in HSC2 systems, SAM/PCI slots 4 to 63 do not exist Furthermore the Jetta HSC2 does not support duplexing so it will not respond to paired space accesses either.

Table 6. lOBus compatible memory space slot and offset bit positions

a. paired/ non paired bit

29 December 1995 28

Polo Software Programming Guide Stratus Company Confidential

5. Xbus/Golfbus PCI to Xbus/Golfbus Address Mapping

The hardware takes a PCI generated address which we refer to as an IOVA and maps it to a system address.

5.1 Software View of Re-mapping

Figure 15 shows the software view of re-mapping. Figure 15. IOVA to System Address Mapping

' re > remote coherent (see Gotfous Functional Spec.cation lor details)

5.2 Polo Re-mapping

Polo supports both an addressing scheme compatible with Jetta based processors, 32 address mode, and an addressing scheme compatible with Runway based processors, 40 bit mode. The sequence starts with a PCI Address or IOVA. Gambit takes the IOVA and sends it across the Xbus. Cyclops re-maps the address to a system address and sends it to the CPU via the IBUS. Figure 16 illustrates the address re-mapping for the 32 bit address mode used by Jetta systems and currently used in Polo systems. The line offset shown in the system virtual index of figure 16 is the line offset taken from the Xbus physical address driven during the Xbus IOVA cycle.

HSC2 supports only the 32 address mode compatible with Jetta based processors. The sequence

29 December 1995 29

Polo Software Programming Guide Stratus ompany Confidential

starts with a PCI Address or IOVA. Mirage takes the IOVA and re-maps the address to a system address and sends it to the CPU via the Golfbus. Figure 16 illustrates the address re-mapping for the 32 bit address mode used by Jetta systems and currently used in Polo systems.

Figure 16. Polo System Address generation (32 bit address mode)

System

Virtual

Index

System

Physical Address

Figure 1 illustrates the address re-mapping used when 40 bit address mode is selected (Polo only). This is the addressing mode that will be used by Runway systems and is included for upward compatibility purposes. Bit 2 of the ASIC specific configuration register within the Cyclops selects between 32 bit and 40 bit address mode. The mode bit is driven on bit 31 of the virtual address to indicate which scheme is being used.

29 December 1995 30

Polo Software Programming Guide uatus Company Confidential

Figure 17. Polo System Address generation (40 bit address mode)

System Virtual Index

System Physical Address

5.3 PCI Memory and I/O Address Mapping

The PCIB/HSC2 has no local memory space. All requests between the Xbus/Golfbus and the PCI adapter are passed through the Gambit Mirage ASIC. The address mapping between the two buses can be best described according to the direction of the request.

5.3.1 PCI to Xbus/Golfbus I/O Requests

I/O Devices have the same view of the whole 4 GB Address Space as the boards on the Xbus/ Golfbus do. Currently the need for a PCI adapter to generate PCI I O space accesses is not apparent The Gambit/Mirage will issue a maintenance attention if a PCI card does an I/O space access. This will happen through the Master Abort mechanism. The Gambit/Mirage will also capture the address on the PCI bus which caused the master abort.

29 December 1995 31 to

Polo Software Programming Guide Stratus company Confidential

Figure 18. 32-bit PCIB/HSC2 Memory Map

32-bit PCI

System Memory Space System Memory Address Map

5.3.2 PCI to Xbus Memory Requests

The Gambit or Mirage ASIC only accepts 32-bit PCI memory addresses in a 256 MB window (32'hX000.0OO0-XFFF.FFFF). The 256 MB block is movable on 256 MB boundaries as defined by the base address registers in the PCI configuration space. The PCI adapter addresses this 256 MB window with the appropriate Chunk Address which is the high 4 bits of the IOVA. Memory accesses created by the adapters that are not within this 256 MB block are terminated through the Master Abort mechanism. There does exist the possibility of accessing a peers memory range on Polo.

The base address registers for Gambit/Mirage and the PCI adapters must not create a conflict in PCI space. All devices and this 256MB block must not overlap in PCI address space. The hardware cannot detect overlaps so the base registers must be set up correctly. The resulting behavior wilt be unpredictable. The driver must also set up the address translation map using the address table registers described in section 12.6.2 before any access from the PCI adapter can take place. tn a Polo system the I/O virtual address (IOVA) is placed directly on the Xbus. Responding agents on the Xbus are told that the address on the Xbus is an IOVA by the requester's having set a bit in the Xbus TRID field. The responding agents then put the IOVA through their local address translation mechanism to check and generate a standard Xbus address.

In a Jetta HSC2 system, the IOVA to system address translation is done entirely with the Mirage ASIC. The location of the address translation, whether in the Mirage as in HSC2 or in the responding agent as in Polo,- is completely transparent from a software perspective.

Figure 18 shows an example of a PCI memory map with 4 adapters mapped. The Gambit/Mirage PCI base address register is set to 32 ^* h1000.0000 and the limit is set to 32'hlFFF.FFFF. This is the bridges 256 MB window to host memory. There are 1 K 16 byte entries in the translation table

29 December 1995 32

Polo Software F-.ogramming Guide tratus Company Confidential

which are indexed by the I/O Virtual Address generated by the PCI adapter.

5.4 Address Re-mapping

Polo and HSC2 use address re-mapping to protect the system from errors caused by simplexed 1/ O devices. Note that the I/O Virtual Address or IOVA format used by Polo & HSC2 systems differs trom the IOVA format used in BIO and PKIO systems.

In Jetta systems the IOVA is translated into a system address inside the Mirage ASIC on the HSC2 board. The map RAM is also located inside the Mirage ASIC and the Mirage ASIC is self-checking.

In Polo, all system accesses from the I/O ASICs are performed using the IOVA (IO virtual address); these are mapped to the final Xbus style physical virtual index addresses (CPU addresses) by a map RAM in the Cyclops ASIC. Both IOVA and CPU addresses are passed on the Xbus, with a bit in the trid field distinguishing between them.

The PCI adapters access main memory via the Xbus/Golfbus for command, data and status transfers. The address generated by a PCI adapter originates from a non-self-checking source, it has to be checked to protect self-checking (safe) memories.

5.4.1 System Address Generation

There is an I/O Address Table allocated in internal memory. This memory resides within the Cyclops ASIC for Polo systems and on the Mirage ASIC for HSC2 systems. In the table there is an entry for each block allocated for I/O in system memory. An entry does not assume any addressing method beyond the basic 16KB block size. It contains the starting virtual index, the starting physical address, and the 12 bits of the ending physical address of the block. The next field is the PCI slot number. This field is used to verify that only the expected slot can access a particular IOVA entry. Figure 19 illustrates the encoding of this field, it is checked against the encoded slot information in the TRID generated by the Gambit.

Figure 19. PCI Slot [3:0] IOVA Map Field Definition - Polo only

Note that the HSC2 system only uses the PCI Slot [1 :0] bits. PCI Slot [3-2] bits are ignored and are not saved in the Map RAM located inside the Mirage ASIC.

There are another four bits allocated for board specific options and a final bit to indicate validity of the entry. Figure 20 shows an entry example using the Xbus/Golfbus address naming conventions. Each implementation will drop unused bits for this table in hardware to reduce the RAM required.

29 December 1995 33

5

VA s Valid Entry

OPTION BITS • see section 5.4.4 PR s Data Pre-read LK = Lock Cycle SW s Swap 16 or 32 bit Endian IC Incoherent Memory Access

This table will be indeterminate at power on. Software should walk through the table and invalidate each entry. Software then must write a valid entry for each device index before any of its respective lOVAs can be used. A maintenance attention will be generated if an entry is used without the valid bit set.

This table is dynamically managed by the I/O driver on a per command basis. The driver has to set up the I/O address table entry for each block before an I/O command that requires memory access can be initiated. The data block can be any size up to 16 KB. Upon command completion the I/O address table entry has to be invalidated by the driver.

There is a map table entry for each 4K page (there can be up to 1024 re-mappabte 4K pages). The page bits of the IOVA, bits 12 and 13 are used to address the map RAM. If the programmer wants to allocate a page in PCI memory space larger than 4K with one device index there must be a map entry for each 4K page. This scheme allows up to 4 contiguous 4K pages in PCI memory space without forcing them to be contiguous in PA memory space.

The driver software gives the I/O Virtual Address (IOVA) to a PCI adapter with every command. The IOVA is the combination of the device index to the I/O address table entry, the page within the device index and the starting Page Line Offsets[l1 :0] into the block. Figure 21 shows the IOVA tor a 32-bit PCI adapter card. It allows for a maximum of 1024 Entries in the I/O Address Table. The next two sections describe the step-by-step process used to create the system address.

Figure 21. 32-bit PCI adapter I/O Virtual Address

31 2827 20 19 1615 14 13 12 11 max.256 Indexes 4 pages 4KB blocks

5.4.1.1 Polo System Address Generation

The Gambit will use the IOVA as is in an Xbus request cycle. The Cyclops in turn will take this information and pass it to the address mapping logic where it applies the appropriate algorithm to perform the checking and address mapping.

The I/O device driver goes through the following steps to set up addresses for an I/O block transfer:

Finds an available slot in the I/O address table.

29 December 1995 34

Polo Software Pi ugramming Guide ^ atus Company Confidential

• Sets up the entry for the block in the I/O address table.

• Sets up the I/O Virtual Address (IOVA) (including the generation of the checksum).

When a PCI adapter initiates a transfer, the Polo system goes through the following steps of address checking and re-mapping:

• Gambit ASIC gets the I O Virtual Address.

• Gambit checks that the access is to the Xbus memory range. If there is no match the access is halted and the error state is generated. The Gambit does a master abort and causes a maintenance attention pm the Xbus.

• Gambit checks the checksum within the IOVA and will cause a Target Abort if it is not OK. This will also cause a maintenance attention.

• Gambit issues an Xbus cycle using the IOVA.

• Cyclops checks the IOVA against the I O address table entry for the device index and page. If the valid bit is set and the lOVA's page and line offset checks out against the entry's start and stop addresses, the combination of the entry and the lOVA's Line Offset[l1:0] create the system address. The Cyclops then looks at the system address to determine if the cycle was directed to its memory space and behaves accordingly.

5.4.1.2 HSC2 System Address Generation

The Mirage will use the IOVA to perform the address map look-up. The address generated from the look-up will in turn be used to generate the Golfbus address.

The I/O device driver goes through the following steps to set up addresses for an I/O block transfer:

• Finds an available slot in the I/O address table.

• Sets up the entry for the block in the I/O address table.

• Sets up the I/O Virtual Address (IOVA) (including the generation of the checksum).

When a PCI adapter initiates a transfer, the HSC2 system goes through the following steps of address checking and re-mapping:

Mirage ASIC gets the I/O Virtual Address.

Mirage checks the checksum within the IOVA and will cause a Target Abort if it is not OK. This will also cause a maintenance interrupt.

• Mirage checks the IOVA against the I/O address table entry for the device index and page. If the valid bit is set and the lOVA's page and line offset checks out against the entry's starting and ending addresses, the combination of the entry and the lOVA's page and line offset[l1 :0] create the system address. Otherwise the access is halted and the error state is generated.

5.4.2 IO Address Table Entry Invalidate

Upon completion of the I/O, the driver software has to invalidate the corresponding I/O address table entry in the table. The table's entry is invalidated by simply clearing the valid bit. The entry cache's valid bit can be cleared by using the address table registers described in section 12.6.2. If this index matches the entry cache's index, the valid bit is cleared.

5.4.3 Address Error Checking

Three other address checking features are employed in addition to the index/entry validity check

29 December 1995 35

3 3>

Polo Software Programming Guide Stratus company Confidential

as described in the next two sections.

5.4.3.1 IO Device Address Checksum

Even though the IOVA is re-mapped it is still used to index into the I/O address table. This leaves a small possibility of an erroneously generated IOVA indexing into a valid, but incorrect table entry. To reduce this probability there are four bits of checksum allocated in the IOVA to cover the index. This number is generated by the driver software when setting up the I/O Virtual Address. It is checked by the Cyclops when the I/O device presents the IOVA for data access.

Supporting the 16K page size in PCI memory space impacts the fault tolerance. In order to keep the 4-4K pages that make up the 16K page contiguous in PCI memory space, the 2 pages bits of the IOVA bits [13:12] are not checked under the checksum. In the case of an error this could allow a controller to wnte to the wrong 4K page within the 16K page. If you want coverage on these two IOVA bits then only make one 4K page valid in the 16K page. Also note that this reduces the number of usable table entries to 256.

The algonthm to generate checksum (C) on the index (I) is the following- C[0] = I[0] ^Λ I[4] ^Λ IOVA[1 ] C[1] = l[1] ^Λ l[5] ^Λ IOVA[15] C[2] = l[2] ^Λ l[6] C[3] = 1(3] * l[7]

This code will detect all single btt errors and all adjacent double, triple, and quad bit errors. Over all the code detects 94% of all possible errors in the 8 bit index (table 7). Note: that the logic exclusive ORs IOVA bits 14 and 15 into the check bits since they should be zero, this does not change the outcome but makes insures that the addresses don't increment out of the 16K page.

Table 7. 4 Bit Parity On 8 Bits Fault Detection Summary

5.4.3.2 Out of Bound Access Check

The ASIC performing the address re-mapping checks to make sure that the physical address [112} of the I/O virtual address is larger or equal to the starting and smaller or equal to the ending physical address [11.2] of the table entry on both read and write accesses. Thus checking is done on 32 bit boundaries. This check is automatic. If you don't want checking, set the starting and

29 December 1995 36

Polo Software . jgramming Guide stratus Company Confidential

ending addresses at their limits.

5.4.4 Control Bits

There are five control bits in the address table entry as illustrated in figure 20 and descnbed in the following sections.

5.4.4.1 Valid Entry bit

The valid entry bit in the table entry indicates whether the entry is valid or not The table should be fully invalidated by initialization software at power on. This involves resetting the valid bit for each of the 1024 entries in the table. When an IOVA is mapped software must write a corresponding entry into the IO map table with the proper system addresses and set the valid bit. tf a device uses an IOVA with an invalid entry the Xbus request will be no-ACKed resulting in the Gambit performing a target abort and generating a maintenance attention.

5.4.4.2 Data Pre-read

This bit is ignored on writes.

In a HSC2 system the Mirage ASIC will perform a data pre-read from the host system if this bit is set and the PCI adapter is performing a read. The Mirage will begin by reading a cache line for the current request and then continue to read successive cache lines until the pre-read is completed. The data pre-read will continue until the pre-read limit in the Mirage is full, the block ending address is reached, or a cycle is issued from the PCI adapter that is not to the block being pre- read. In the event of a non sequential address the pre-read data will be flushed. If this bit is not set then the Mirage will only read the 32 bit memory word requested by the PCI adapter and nothing else until an additional request is made. We may make use of the PCI commands read and read- multiple.

In Polo, all memory accesses through the PCIB initiated by a PCI adapter are issued as 32 bit reads on the Xbus. A full cache line is returned if pre-read is on, lock is off, the starting address is on a cache line boundary, and the ending address of the cache line is less than the block ending address (if the starting address of the access is out of bounds then the access is illegal and not responded to). If all of these conditions are not met on a legal access then the Cyclops will only return 32 bits of data. The Polo system will only use this bit as a qualification for returning either 32 bits or a cache line on a read. The Gambit ASIC will accept either a word or a full cache line back in response to a cache line read request.

5.4.4.2.1 Coherency Restrictions software note:

Improper use of the pre-read option can result in inconsistent data in the system. It is software's responsibility to guarantee data coherency by insuring that the PCI adaptor never receives "stale" data from a pre-read buffer.

On an HSC2 system, pre-read data sits in the Mirage until one of the following conditions are met:

The data has been completely read by the PCI adaptor

The host updates the Map RAM and the map table entry matches the Map cache entry being used by the Pre-read state machine.

29 December 1995 ^ 37 S

S97/09781

Polo Software Programming Guide Stratus company Confidential

• The host explicitly flushes out the pre-read buffer by writing to flush pre-read data bit in the Test Control Register.

Note that PCI adaptor writes while pre-read operations are taking place do not cause data to be flushed from the Mirage even If the write transaction is to the same page as the data pre-read.

5.4.4.3 Lock Cycle

This bit is ignored on writes.

The data pre-read bit is ignored and should be set to zero if the lock cycle bit is set to one.

In a HSC2 system any read access to the block will go out onto the Golfbus with a lock cycle function code (a loadclear operation) if this bit is set. The data pre-read bit is ignored and should be set to zero if the lock cycle bit is set to one.

In a Polo system the I/O address map RAM resides on the other side of the Xbus so the cycle will not have a lock function code when it is on the Xbus. However if the I/O address map look-up reveals that the lock cycle option bit is set then the Cyclops will turn the cycle from a cache line read into a load/clear operation (semaphore lock). The Gambit ASIC will accept either a full cache line or a 32 bit word back on a read access so the Cyclops will only send back the 32 bit word provided by the load/clear. The data pre-read bit is ignored and should be set to zero if the lock cycle bit is set to one.

Note: The swap bit must be turned on when using the lock bit to avoid the possibility of Inconsistent swapping between two boards in a duplexed pair.

5.4.4.4 Swap 32 bit Endian

When this bit is set to a one the data associated with this IOVA address will be big/little endian byte swapped. If this bit is set to a zero then there is no byte swapping is performed. Refer to section 5.6 for a detailed description of byte swapping and when this bit should be set

5.4.4.5 Incoherent Memory Access

This bit allows software the option of generating incoherent (don't snoop, dont set remote tag) as opposed to coherent (snoop, but don't set remote tag) Xbus/Golfbus memory read and write transactions. Refer to the section titled Remote/Coherent bits in the Xbus Golfbus specification for more details.

• 0 - coherent 1 - incoherent

In a HSC2 system, if this bit is set to a one then a read or write operation will go out on the bus with the remote coherent bits set to indicate an incoherent operation. If this option bit is set to zero then the remote coherent bits indicate a coherent cycle.

In a Polo system the snoop results will be ignored if this bit is set to a one. All reads and writes will be issued from the PCIB with the remote/coherent bits indicating a coherent cycle. The remote/ coherent bits sent across the tbus will reflect the state of this bit.

29 December 1995 , _Λ 38

Polo Software atus Company Confidential

5.5 Software Map Management

The four address table registers that control reading and writing of the I/O address map RAM (see section 12.6.2) will require the use of locks to guarantee exclusive access. There is only one set of these registers accessible from 16 different addresses so a single lock should be used to control them.

5.6 Byte Alignment

The Gambit/Mirage is the interface from the big-endian byte order Xbus/Golfbus to the little-endian byte order PCI Bus. Significant endian issues exist within the PCI Bus as data structures (including SCRIPTS code) are required to be in little-endian format while disk and tape data are required to be in big-endian format (Stratus currently wπtes to disks (K105/K121) and tapes (K116) in big- endian format).

Many data structures (including 82596 SCB's) intermix 32-bιt entities with 16-bit and 8-bιt entities which causes further byte ordering conversion issues

The PCIB/HSC2 system does not attempt to resolve these endian issues, rather it supports a method of moving data into and out of the PCI adapter with the byte swapping under software control. This is configurable on a page by page basis through the option bits in the address map (refer to section 5.4.4.4). It is the responsibility of software to resolve any remaining endian issues.

The Gambit/Mirage will always perform a byte swap on data going from the lOBus/Xbus to the PCI bus. When going from the PCI adapter to the Xbus the byte swapping is software programmable via an option bit in the system IO address table re-mapping entry. Figure 22 represents the byte swapping performed for a little/big endian byte swap. A byte entering a specific byte lane (i.e. byte 0; big-endian) on one side (Xbus or PCI bus) will be sent back out on the same byte lane (byte 0; little-endian) on the other side. Thus when a 32 bit word is byte swapped its format changes from big to little endian or vice-versa so that the data is identical from a software perspective when viewed from either the little endian (PCI) or big endian (Xbus) side of the system. This is the default configuration and the one which should be used when moving data from to disk and tape devices. Programmable byte swapping is not supported for requests coming in from the Xbus/ Golfbus to the PCI bus - data is always byte swapped so that it appears the same in either format to software.

29 December 1995 39

31

Polo Software Programming Guide Stratus company Confidential

Figure 22. Little/Big Endian Byte Swap r

r

Figure 23 represents the byte swapping performed when the option bit in the address field is configured for no endian swapping. A byte on bits 7:0 traveling from the PCI Bus to the Xbus/ Golfbus will remain on bits 7:0. tn this mode an integer will hold the same value whether it is in PCI memory or in system memory.

Figure 23. No Little/Big Endian Byte Swap increasing order of significance

Gotfbus/Xbus data bits _ 31 2423 16 15 87

Big-Endian byte O byte l byte 2 byte 3 (Golfbus/Xbus) T

PCI bus data bits . 31 2423 L 1615 87

Little-Endian byte 3 byte 2 byte l byte O (PCI bus) increasing order ot significance

The following table illustrates the byte written or read based on the system bus byte enables. Note that the big endian point of view is the standard system bus point of view and the one that should be used when referencing the byte enables.

29 December 1995 40 ^"

Polo Software hugramming Guide _^ ..atus Company Confidential

a. NC - no change, wntes - this byte holds the previous value; on reads this data is not valid.

Table 9 illustrates how various HSC2/PC1B data cycles are affected by byte swapping. Table 9. Big/Little Endian Byte Swapping Based on Access Type

29 December 1995 q 53c. 41

97/09781

Polo Software Programming Guide Stratus company Confidential

6. Interrupts

Interrupts generated by cougar, ReCC, and Sable remain unchanged and are well specified in the appropriate specs. This section defines the interrupt structures associated with the PCIB and HSC subsystem.

6.1 PCIB/HSC2 Interrupts

6.1.1 Introduction

The PCI bus specifies the mechanism through which PCI devices can interrupt the host The PCI interrupt mechanism defines 4 level sensitive interrupts, (INTA, INTB, INTC, and INTD). Three of these interrupts (INTB, INTC, and INTD) can only be used by multi-functional devices. Non- multifunctional devices must use INTA. The interrupt output structures are defined to be open drain signals with no restrictions on grouping together or interrupts. The implementation only supports the one INTA# interrupt (actually a combination of att four lines tied together in hardware). This saves 3 pins and only marginally complicates software for multifunctional device drivers.

The HSC2/Polo interrupt scheme is based around the method that HP uses to implement interrupts on the PA-RISC chip. The HP scheme is a transaction based interrupt architecture. Interrupts are signalled by setting a bit in the EIR (External Interrupt Register) inside the PA chip. In HSC2 Polo, interrupting devices may select to interrupt a specific processor or all processors by writing to either one or all EIR registers.

6.1.2 Implementation goals

The specific goals of the PCIB/HSC2 interrupt scheme are:

Directed Interrupt Capabilities

It should be possible to select the target host on a per-PCI adapter basis. This allows for selection of either a particular CPU or all CPUs on an individual SAM basis.

• Minimal host overhead

The host should not have to poll SAM modules in order to determine the source of the interrupt.

• simultaneous interrupt support

Since multiple SAMs may share the same EIR bit, any implementation must be able to support simultaneous interrupts. Schemes that depend on only one interrupt being presented until that interrupt is serviced are not acceptable.

• Simple

Complexity in either hardware or software is undesirable.

6.1.3 Hardware Implementation

In Polo/HSC2, there are some severe system limitations on how interrupts are managed within the EIR register ot the PA chips. Each PA can have 32 interrupts, but of these only a small number can be allocated for a given slot. Without some Kind of additional hardware support, it would be necessary for the interrupt handler to poll all PCI adapters on the interrupt. This would not make for very optimal performance.

26 December 1995 42

Polo Software t Company Confidential

For that reason, the hardware must implement more than a write to an EIR register to perform an interrupt. In order to prevent polling of all the PCI adapters, each SAM writes an interrupt into the EIR register and to an OS specified byte (the Host interrupt table) in cacheable memory. The OS can specify any location in memory for this byte. This allows the OS a great deal of flexibility. The OS can then scan the table of interrupts at memory speeds to determine which PCI adapter interrupted it In the suggested software implementation, some possible architectures for this table are discussed.

In order to implement this scheme, the hardware makes use of a series of registers, operations, and memory tables. The software team should evaluate this proposal to ensure that it is both necessary and sufficient for their needs. Six registers have been implemented on the SAM for hardware support of interrupts. See the corresponding register descriptions in section 12.6.4 on page 129 for a bit-by-bit description.

SAM Interrupt Source RegisterfO]

This read only register reflects the state of the interrupt pins on the PCI bus. The signal is high true and used to trigger interrupts. This bits reflects the synchronized value of the interrupt signals on the PCI bus.

SAM Interrupt Mask Register[0]

This register is used to enable and disable individual interrupts in the interrupt source register. Asserting a bit in this register enables interrupts in the interrupt source register to cause an interrupt event

SAM Host Interrupt Table Pointerf47:0]

This 48 bit address points to a location in memory that will be used as a table entry to identify the interrupt source.

SAM Host Interrupt Address Pointerf.31.2]

This 32 bit I/O address identifies an EIR register address. The interrupt bit register value will be written into this register.

SAM Host Interrupt Bit Register[5:0]

Bits [4:0] of this register point to the bit to be set in the host interrupt register. EIR and bits 5:0 are written to the interrupt table pointer address. Bit 5 is always a one and functions as a valid bit for the table entry.

Interrupts are generated by either of two conditions:

Condition 1: If there is a zero to one transition on the interrupt bit, and the interrupt mask bit is set to one, an interrupt action will be triggered.

Condition 2: If there is a zero to one transition on the interrupt mask bit, and the interrupt bit is set, an interrupt action will be tnggered.

The hardware performs an interrupt operation in the following manner:

Step 1: A byte write is performed to the address indicated by the SAM Host Interrupt Table Pointer. This write is performed as a cacheable memory write, so the software must use a valid memory address for this location, not an I/O address. The data pattern for the wnte is taken from the least significant byte of the SAM Host interrupt bit register. No validity checks are performed on the wnte.

26 December 1995 43

Polo Software Programming Guide Stratus .npany Confidential

Step 2: A 32 bit I/O write is performed to the address indicated by the SAM host interrupt register pointer. The data pattern written is all zeros except for the bit indicated by the SAM host interrupt bit register. It is intended that the software set the address ot this write to point to either an individual Cougar EIR register or to all CPUs' Cougar EIR register.

No further interrupts will be transmitted to the host by this SAM until one of the 2 conditions listed above is met

These two wntes are not atomic. Software running on the PA can make no assumptions concerning timing of these two writes. Furthermore, it is possible for two SAMs to perform these operations at the same time. When this occurs, any mixing of wnte operations is possible between the SAMs. If there is a common interrupt bit shared by the two SAMs, software must ensure that it correctly decodes all of the interrupts. However, the writes are strong ordered on a per SAM basis, that is the two wntes from a particular SAM are guaranteed to be seen in the correct order.

For these reasons, it is suggested that software follow the algorithm below for decoding and handling interrupts.

6.1.4 Suggested Software Implementation

As part of the PCI interrupt handling mechanism, the OS must manage the host interrupt table which consists of a one byte addressable piece of information per PCI adapter that is used by the hardware. Since this table is in cacheable memory software must keep the virtual index up to date. This is probably best accomplished through static allocation of the table. Since the SAM register that contains the pointer to this byte ts justified down to the byte, the software has a great deal of flexibility in how they organize the host interrupt table. While there are a number of possible organizations, the suggested implementation is to use one host interrupt table per system. This table should be organized as 64 contiguous bytes of PCI interrupt information. This organization allows for the fastest possible scan of the interrupt table. Optimally, all interrupts should be directed to one of the CPUs local to the host interrupt table. That CPU can perform the steps below and then delegate the servicing of the interrupt request to the appropriate CPU.

The order in which various operations associated with the interrupt handling are performed is also important. Deviations trom the suggested implementation may result in either extraneous null interrupts or lost interrupts. The basic steps are outlined below:

Step 1 : interrupt is received

Step 2: internal interrupts are masked.

Step 3: the appropriate bit in the EIR register is cleared.

Step 4: The host interrupt table is harvested for any pending interrupts.

Step 5: At this point, or later in the sequence, internal interrupts can be unmasked.

Step 6: Gambit or Mirage interrupt mask register is set to 0 (disabling Gambit or Mirage interrupts).

Step 7: PCI card specific actions are performed to clear the interrupt.

Step 8: Gambit or Mirage interrupt mask register is set to 1 (enabling Gambit or Mirage interrupts).

Step 9: The interrupt is serviced.

26 December 1995 44

Polo Software i .^gramming Guide »ratus Company Confidential

7. Boot

This section of the spec outlines the boot process tor booting an OS on either a Polo or HSC2 machine. The boot process consists of the operation of loading an OS image into main memory and beginning normal OS execution. Before describing the boot methodology in detail, the boot alternatives that have been considered will be described.

7.1 Boot Alternatives

PCI boot is a challenging opportunity for both systems. Since FTIO will be offered in an MIO only configuration (i.e. no BIO) and Polo's only configuration is a PCI only configuration. A methodology for booting the OS in a PCI only configuration must be supported. A number of possible approaches for accomplishing this task are outlined below.

7.1.1 Baby BIO Boot

The Baby Bio boot alternative assumes that Stratus commits to designing a BIO replacement with a PCI interface. If Stratus follows that approach, then the Baby BIO would be the only bootable PCI device and all boot functions would be performed by the Baby BIO.

The major trade-off in considering boot altematives is the balance of hardware effort and software effort. A baby BIO approach to boot represents a trade-off that minimizes software effort at the expense of hardware. In this approach, a PCI card would be designed that mimics that BIO functionality but has a PCI interface instead of a Golfbus interface. Much of the BEHI and BIO software could be recycled, and changes to the boot prom would be minimized. The baby BIO would respond to a boot command in the same fashion that it does today.

A fundamental problem with this approach is that it probably could not be ready in time for initial power on.

7.1.2 Bio Firmware on Raid Controller Boot

This is a variation on the Baby BIO boot. In this alternative, Stratus would buy an i960 based raid controller with 720 SCSI scripts processors and port the Bio firmware to it. In this approach, most/ all of the firmware would need to be re-written, but the CPU Prom and the BEHI could be maintained.

This approach has the advantage of minimizing the hardware and boot prom effort. However, it is not clear if this card exists. Also, the firmware effort would be very large.

7.1.3 CPU Prom Resident PCI Card Boot

In this boot alternative, the CPU PROM would contain all information and algorithms necessary to perform a boot of either OS off of a given PCI card. Given that PCI cards are not very intelligent, this may be a substantial amount of code. The advantage of this approach is that the system would be able to boot off of any PCI device. There are several disadvantages to this approach. Stratus would have to write PA7100 specific code for each boot device in the system. Every time a new bootable device is released, a new boot loader would have to be written and qualified. Also, every time that a new card is released, a new boot prom would have to be released as well.

26 December 1995 45

Polo Software Programming Guide Stratus company Confidential

7.1.4 Expansion ROM Resident PCi Card Boot

A variation to the CPU Prom resident boot is to use a PCI card resident boot. This method makes use of the expansion ROM on each PCI card. PCI cards have an expansion ROM that can be used to store self-test, boot, and other code. Stratus could write PA7100 boot strapping code for each PCI card that is supported that contains the required boot code. The CPU prom would only need to know how to perform a generic ROM copy and then execute that code. This approach requires a small modification to the CPU boot PROM, but it would require every new bootable PCI card to have a special boot driver written for it. This work is expected to be substantial.

7.1.5 x86 Emulator PCI Card Boot

A third variation on the boot from PCI card theme is to make use of not only the expansion ROM on the PCI card, but the vendor provided boot code as well. This could be done by supporting an x86 emulator in the CPU PROM. This emulator would then run the PCI BIOS code directly. The advantage to this methodology would be that the installed base of PCI card code is primarily x86 code, and this would allow us to make the maximum use of that code. However, it would take considerable work to support the emulator. Also, hardware changes must be made to support such features as the DOS compatibility hole. Finally, the address mapping scheme may have to be disabled to allow for some address limitations in the PCI code.

7.1.6 PCMCIA Flash PROM Based Boot

A totally different variation on boot is to choose to never boot off of a PCI card. In order to accomplish this boot method, a dedicated Flash PROM card would be used instead. This functionality would be duplicated and located on a PCMCIA bus on each repeater in MIO and the 2 PCIB cards on Polo. The CPU Prom would configure the PCMCIA bus and then initiate the copy of the boot-loader or OS into main memory. This approach has the advantage that the number of boot devices that need to be supported is minimized. The boot process is kept simple. It has the disadvantage that it requires some specialized hardware. Also, some new utilities will have to be written to manage the OS images and the flash prom.

7.1.7 Conclusions

Table 10. summary of pros & cons

26 December 1995 46

Polo Software Programming Guide buatus Company Confidential

Table 10. summary of pros & cons

The table above lists the potential pros and cons of each solution. The standing proposal is to use the flash prom boot. This decision is based on the belief that the 2 closest altematives to the flash prom boot are the baby bio and the expansion prom boot The baby bio minimizes the software effort at the extreme expense of the hardware effort While the expansion prom boot minimizes the hardware effort at the extreme expense of the software effort The flash prom alternative seems like a good compromise.

7.1.8 Open Boot

A note on Open Boot- Open boot is a proposed standard to allow for a processor and O/S independent boot process. It is currently unclear what direction this process is taking. We have currently chosen not to count on open boot being available in the initial Polo/HSC2 time-frame. However, should this change, it may be advantageous to reconsider some of our current directions.

7.2 PCI Boot Process

7.2.1 Find and Configure the Boot SAM

Once the PROM has accomplished its normal initialization and is ready to boot the system, it checks the boot list in the ReCC NVRAM. There should be two SAMs identified in the boot list as primary and secondary bootable devices. The SAMs and PCI buses for the devices need to be configured before they are used. The following steps are followed to do this:

1 ) Set the HSC2 PCIB on-line by writing to the HSC2/PCIB bus interface state register.

2) Set the SAM on-line by writing to the SAM bus interface state register.

With both the PCIB and SAM on-line, PCI configuration cycles can be run to initialize the PCI bus.

The PCI configuration cycles are accomplished by first writing to the PCI Configuration Address register and then writing/reading to/from the PCI Configuration Data register. For Configuration cycles to the bridge, the enable bit (bit 31 ) should be enabled, the bus number (bits 23:16) should be 0, the device number (bits 15:11 ) should be 0, the function number (bits 10:8) should be 0, the register number should be set to the register number tor the desired configuration register, bits 1

26 December 1995 47

Polo Software Programming Guide Stratus ^umpany Confidential

and 0 should be 0.

WARNING: Configuration addressing is little endian. See the endian description for a discussion of how endian works. The summary is that byte operations work with no intervention, 32 bit operations must be byte swapped!

1) Perform a 32 bit read from configuration address 0. Verify the manufacturer ID, 16 bit word at location 00, is H07h. Verify the Device ID, 16 bit word at location 02, is 0700 for IAM or 0600 for Gambit

2) Perform a 16 bit write to location 04 of I6'h127. This write disables fast back-to-back enable, enables SERR, disables ECRO, disable address stepping, enables parity error responses, disables VGA palette snooping, disables memory write and invalidate cycles, enables special cycles, enables bus mastering, enables memory space and enables I/O space.

3) Perform a 16 bit write of location 06, the status register, verify that it is 0.

4) Set the memory base address register to 0 by performing a 32 bit write of Oh to address lOh.

5) Set the disconnect timer to 0 by performing a byte write of 0 to 40h.

6) set the retry abort timer to 8 by performing a byte write of 8 to 41 h.

7) set the trdy timer 1020h by performing a byte write of 20h to 42h. The bridge is now configured.

7.2.2 Configure PCMCIA Bridge Chip

Once the PCI bridge is configured it is necessary to configure the PCMCIA bridge chip by performing a series of configuration reads and writes to the PCIMCI bridge. For these writes, the enable bit (bit 31) should be enabled, the bus number (bits 23:16) should be 0, the device number (bits 15:11) should be 1 , the function number (bits 10:8) should be 0, the register number should be set to the register number for the desired configuration register, bits 1 and 0 should be 0. The PCMCIA bridge chip (PPEC) is a multifunctional device. However Polo only uses the first function.

1) Perform a 16 bit read of register OOh. Verify that the Vendor ID is 8086h, Intel.

2) Perform a 16 bit read of register 02h. Verify that the device ID is 1221 h, ppec chip.

3) Perform a 16 bit write of I4bh to location 04h. This write disables fast back-to-back enable, enables SERR, disables ECRO, disable address stepping, enables parity error responses, disables VGA palette snooping, disables memory write and invalidate cycles, disables special cycles, disables bus mastering, enables memory space and enables I/O space.

4) Perform a 16 bit write of cβOOh to location 06h.

5) Initialize the PCI-PCMCIA bridge base address register. This register determines the starting address of the PCMCIA index/data socket configuration registers in the PCI I/O space. It is recommended that 0 is used is for the base address. This is accomplished by writing a 1 to location 10h. After boot this base address can be deconf igured or moved out of the way for normal operation.

6) set the PCI Configuration control register. Perform a byte write of 39h to location 40h. This

26 December 1995 48

Polo Software F . -gramming Guide _^ ..atus Company Confidential

enables enhanced PCMCIA timing mode, enables PCMCIA read-prefetching buffers, enables write posting buffers, set the PCI clock speed to 25Mhz.

7) set the PCI interrupt routing register. Perform a byte of 3h to location 50h. This routes both of the PCMCIA interrupts to the interrupt line and disables interrupts from all other (unused PCMCIA) sockets.

The PCI side of the PPEC chip should now be configured.

7.2.3 Configure PCMCIA Bus

The PPEC chip is run in modeO. This operational mode provides two fully buffered PCMCIA sockets. Polo and HSC2 only populates socket A so all accesses should be performed to socket A register and windows. Like the PCI side of the PPEC, the PCMCIA side requires some initial set up. The set up is accomplished through performing I O writes to the PCI bus. The PCMCIA chip uses an index port and a data port to access its registers. The index port is a byte located at the base address loaded into the base address register in step 5 above. The data port is a byte located at base address + 1. If the algorithm was followed as outlined in step 5, then the base address is at 0 in PCI I/O space. If the algorithm was not followed, then you must do your own translation. Writes can be performed to the PPEC chips by performing a 16 bit write with bits 7:0 corresponding to the desired register and bits 15:8 corresponding to the data. Reads may be performed by performing a byte write to the index port and a byte read to the data port.

WARNING: this register and all descriptions are little endian. Byte swapping must be performed if anything other than byte operations are performed.

1) Configure the PCMCIA power. Perform a 16 bit write of 1000h to PCI I/O space Oh. This will disable the output enable for the PCMCIA bus, turn off auto-powering, turn the socket on to 5v, and turn VPP on to 5v.

2) Enable the PCMCIA bus. Perform a 16bit write of 9000h. This write enables the PCMCIA output enables for actual writes to the PCMCIA bus.

3) Set up the I/O window data side. The PCMCIA card has an 8 bit I/O bit data path. This is configured by performing a 16 bit write of 1007h.

4) Setup I/O windowO. The PPEC chip contains two I/O space windows. The I/O space is passed directly through the PPEC chip. I/O accesses are used to setup and control component management registers starting at address 4000h. We will setup 4000h through 4lffh as the I/O space for the PCMCIA bus. These may be deconfigured after boot. Set the low byte start address for I/O window 0 by performing a 16 bit write of 4008h. Set the high byte start address for I/O window 0 by performing a 16 bit write of 0009h. Set the low byte stop address by performing a 16 bit write of ffOah. Set the high byte stop address by performing a 16 bit write of 4l0bh.

5) Setup Memory Address Mapping Window 0. The PPEC contains 5 memory windows. The memory windows are used for accessing the command register and the flash array itself. For simplicity's sake we will map this address space starting at 0 and running through the full 20MB range. After boot this address can be deconfigured. The window allows for a 16Mbyte region of PCI address space to be mapped at one time, and for the full 64Mbyte region of PCMCIA space to be mapped, the mapping granularity is a 4K page. We will setup a window of 16Mbytes of PCI space and l6Mbytes of PCMCIA space and then reset the windows when we roll over iβMbytes. Set the system memory address mapping window 0 start low byte (bits 19:12) by performing a 16 bit write of 001 Oh. Set the system memory address mapping window 0 start high byte (bits 23:20) by performing a 16 bit write of 8011 h. Bit 7 of this write sets the data path to 16 bits, and bits 3:0

26 December 1995 49

Polo Software Programming Guide Stratus ompany Confidential

set address bits 23:20 to 0. Set the system memory address mapping window 0 stop low byte (bits 19:12) by performing a 16 bit write of ff12h. Set the system memory address mapping window 0 stop high byte (bits 2320) by performing a 16 bit write of 6fl3h. Bits 7:5 set the timing of the card to 150ns. Bits 3:0 set the upper address to f. Setup the PCMCIA card offset address. This will initially be 0. We will bump this up when we roll over the 16Mbyte region. Set the card memory offset address 0 low byte register by performing a 16 bit write of 0014h. Set the card memory offset address 0 high byte register by performing a 16 bit write 0015h.

6) Enable the address mapping windows. Perform a 16 bit write of 4i06h. This enables I/O window 0 and memory address window 0.

With these settings:

I/O addresses 0 and 1 map to the PPEC chip.

I/O addresses 4000h - 4iffh map to PCMCIA I/O space

Memory addresses OOOOOOOOh - OOffffffh map to PCMCIA space.

7.2.4 Configure PCMCIA Flash Card.

The PCMCIA flash card is accessible as a slave on the PCMCIA bus. The card is organized into an attribute space accessed through I/O space and a memory space accessed through memory space. The first step is to configure the I/O space to the proper configuration.

1) Configure the voltage register by performing a byte write of 01 h to 4000h. This sets VCC to 5v and VPP to undriven.

2) turn on VPP generation by performing a byte write of 01 h to register 410ch. This turns on the internal VPP generator in the PCMCIA chip.

The card is now ready to be read for boot. This operation is started by performing a command write. The command wnte sets the mode of the flash array. Perform a 16 bit write of hex ffff to memory space Oh.

The card is now in read mode and memory reads will be processed directly to the array. The array may be read in 32 bit chunks.

Notes:

This description did not describe setting up the Gambit/Mirage offset window registers to the PCI bus. These must be setup.

This description did not explicitly explain the reprogramming of the card offset register when the prom rolls over the 16Mbyte PCI page.

Writing the PCOM is not described, and will be covered in a later revision of the spec.

26 December 1995 50

Polo Software Pi gramming Guide ,atus Company Confidential

8. PCI Configuration

The Polo and HSC2 PCI bridge and adapters have to be configured. This will need to be done at boot time. There are also configuration issues that need to be addresses when new devices are plugged in, pulled out or when they fail.

PCI is defined to allow the PC to do an auto configuration. The PC BIOS discovers which devices are on the PCI bus and configures the bridges and adapters. (There is some user intervention required in some cases where it cannot be completely automatic - see PCI ECR Status document dated 5/11 94.) When a configuration is complete the information is stored in non-volatile memory and is used in subsequent boots as long as the same devices are present

We need to support a configuration similar to the auto configuration done in PCs. Cards may have swapped since the last boot We also need to support hot-plug reconfiguration due to board pseudo-hot plugs in the Polo and HSC2 systems.

8.1 PCI Configuration Register Space

Each PCI agent, including the host bridge within the Gambit/Mirage ASIC, has a 256-byte configuration space. The first 64 bytes of this configuration space comprises the required PCI configuration registers. These are a predefined header containing ID, status, command and other information. The optional configuration space from 040h-FFh is available for use by bridges. Gambit/Mirage will not use this area. All PCI adapters/agents must respond to the entire 256 byte configuration space. All writes to undefined areas must have no effect and all reads to undefined areas must return zeros.

The Gambit/Mirage ASIC implements accesses to the configuration space using a method similar to the PC/AT. Accesses to the 32 bit I/O registers, config_addr and the conf ig_data, defined in the Gambit Mirage I/O register space are used to generate PCI cycles with the Configuration Read or Write function code. First an address is set up by writing to config_addr then a read or write is done to configjdata. See configuration section 8.5, PCI Address formation, for specifics on how the conftg_addr decodes to the individual PCI devices on the PCI bus.

8.1.1 Gambit PCI Configuration Space

The PCI configuration space of Gambit Mirage is not available to other agents on the PCI bus, but only to Xbus generated requests. Accesses to these registers are made through the 'config_addr ^* and 'configjdata' registers in Gambit IO Register Space. The Gambit PCI Configuration Space is available as 'Device Number 0' for type O0' accesses.

Table 11. Gambit/Mirage PCI Configuration Space Header

29 December 1995 51 °

Polo Software Programming Guide Stratus Cuinpany Confidential

Table 11. Gambit/Mirage PCI Configuration Space Header

29 December 1995 52

Polo Software .-^ramming Guide _ -atus Company Confidential

Table 11. Gambit/Mirage PCI Configuration Space Header

8.1.2 PCI Adapter Configuration Space

The PCI configuration space of the mated PCI adapter is available to the Gambit/Mirage ASIC across the PCI bus. Accesses to these registers are made through the 'config_addr' and 'config_data' registers in Gambit/Mirage IO Register Space. The PCI Adapter Configuration Space is available as 'Device Number 1 ' for type '00' accesses. For adapter specific details on the Configuration Space Header, refer to the PCI Specification Chapter 6 and the appropriate Adapter Specification.

Table 12. PCI Adapter Configuration Space Header

29 December 1995 53

_ Λ Λ

Polo Software Programming Guide Stratus ,jmpany Confidential

Table 12. PCI Adapter Configuration Space Header

8.1.3 PCI Configuration space header description

These registers are defined in the configuration space for system bus/PCI host bridge and for each PCI adapter in the system.

The Vendor ID field is read in each slot to check for card existence. It is a 2 byte field. The PCI spec, states that if the read returns 16'hFFFF this is an indication that the device does not exist. On Polo HSC2 the Xbus/Golfbus I/O read will be aborted and cleared out of the PING table in Gambit/Mirage. This will result in a PING timeout which will then cause a data return of zeroes with an LPMC to be sent to the Cougar. The OS should search for available cards and not the driver code. Otherwise the vendor ID should be checked for a valid Stratus supported device.

The next field read should be the Device ID. Again it should be checked against known devices for the vendor.

The Revision ID may have meaning in some cases. We need to see what specific adapters do here and what significance the number holds.

The Header Type identifies the layout of bytes 10h through 3Fh and bit 7 indicates if the device is a multi-function device. Type 00 is a normal card and type 01 is a PCI-PCl bridge header format. The other encodings are reserved. If the device is multi-function the all of the function numbers have to be polled for existence. If it is a PCI-PCl bridge there are additional type 01 configuration cycles required to configure the lower level PCI buses

The Class Code field is a 6 byte field, the first two bytes indicate the base class (VGA, Mass Storage, Network, Bridge...) The next two bytes are the Sub-class code (SCSI, IDE, Floppy...). The next 2 bytes are the Programming Interface which indicates a register level programming interface for device independent software.

The Command register control the major functions of the PCI card. Writing all zeros to this register logically disconnects the card trom the PCI bus.

Bit 9 • Fast Back-to-Back Enable - Reset since Gambit/Mirage will not support it

Bit 8 - SERR# enable - This bit will be turned on.

Bit 7 - Wait cycle control - Reset to zero if writable.

Bit 6 - Parity Error Response - Set to enable parity checking

Bit 5 - VGA Palette snoop - Not supported

Bit 4- Memory Write and Invalidate - Not supported

Bit 3- Special Cycles - Reset to zero to disable. Hardware is capable if Software wants to turn this on. Bit 2- Bus Master - Set to 1 Bit 1- Memory Space - Set to 1 Bit 0- IO Space - Set to 1 on adapters, Set to 0 on the bridge.

29 December 1995 54

Polo Software Programming Guide biratus Company Confidential

The Status register

Bit 15 - Detected Parity Error

This bit indicates that a device detected a parity error. A Maintenance Interrupt is generated from Gambit/Mirage. An adapter is responsible to report a parity error to the system most likely via an interrupt to its driver. Bit 14 - Signaled System Error

This bit gets set whenever a device asserts SERR# which is a global error signal directed to the Operating System. SERR# indicates an address parity error, data parity errors on a special cycles or other non parity errors. SERR# will cause a

Maintenance Interrupt to be generated from Gambit/Mirage. Bit 13 - Received Master Abort

Gambit/Mirage causes a Maintenance Interrupt.

Adapter - TBD Bit 12 - Received Target Abort

Gambit Mirage causes a Maintenance interrupt.

Adapter - TBD Bit 11 - Signal Target Abort

Gambit/Mirage causes a Maintenance interrupt.

Adapter - TBD Bit [10:9] - DEVSEL timing

00 - fast

01 - medium

10 - slow

These bits are read only and indicate the speed at which a device will decode accesses to its address spaces. It can be used by the subtractive decode agent to speed up selection. Gambit Mirage does not do subtractive decode so the Master will have to use the Master Abort Mechanism. Bit 8 • Data Parity Detected

This bit is only set by a bus master when: 1. When the master asserts PERR# (on a read transaction) or when a master samples PERR# (on a write transaction) and 2.

The Parity error response bit is enabled. Bit 7- Fast Back-to-Back Capable

The Cache Line Size register - This register only needs to be implemented on devices that are participating in the caching protocol which we do not support.

The Latency Timer- register must be writable on all master devices that can burst more than two data phases. It may be read only on devices that burst two or fewer data phases but must contain the number 16 or less.

The Built-in Self Test (BIST) register

Bit 7 - BIST Capable

Bit 6 - Start BIST

Bits [5:4] - Reserved

Bits [3:0] - Completion code: 0 indicates test passed; non zero device specific failure codes.

The Interrupt Line register - Use of this register is TBD. In a PC it corresponds to the IRQ level of the 8259.

The Interrupt Pin register - This register indicates which interrupt line the device will use. It will contain 0 if no interrupts are used. Polo will tie all interrupts to INTA#.

29 December 1995 55

T/US97/09781

Polo Software Programming Guide Stratus ompany Confidential

The MINJGN register - This register specifies how long a burst period the device needs assuming a 33 MHz clock. This value should be the recommended Latency Timer value or 0 if there are no special requirements.

The MAXJLAT register - This register contains the time this device can wait for a grant. This number is used to set the latency timers of other devices. There must be some algorithm/ equation but we don't know it.

8.2 Proposed configuration sequence

1. Boot system - See Section 7.

2. Start OS based config-for each PCI slot and sub-function a. Device detection - Configure host bridge if it exists b. Match device against requirements - then config the device i. Run the post code on the devices Expansion ROM ii. or Run the cards initialization code stored in the OS files system iii. or Restore the previous configuration (unlikely to be choice, discussed later) c. Shut down any unknown unconfigurable devices d. Update host bridge configuration with new address information. e. Run any diagnostics on device i. Expansion ROM based diagnostics ii. or OS based diagnostics f. Handle special configuration i.e. dual initiated scsi host ids etc. g. Set up device drivers

8.3 OS Based full system Configuration

The PCI config_addr and config_data registers are in Gambit/Mirage and there is one register that responds to the address corresponding to 4 SAMS. They map as follows:

NOTE: To control access to these multi-response register the OS needs to lock the group of 4 SAM registers since they are physically one register.

8.4 PCI device detection

All PCI devices adaptors and bridges must implement the first 4 fields in the header, Vendor ID, Device ID, Command and Status. The additional fields are optional but if used must be at the appropriate memory address.

In a PC all possible PCI slots must be polled to determine the configuration. The polling starts at device 0. If a multi function device is detected the all other functions in the slot must be polled. Our system will avoid polling since each SAM will supply a maintenance interrupt when it powers on or

29 December 1995 56

Polo Software Programming Guide biratus Company Confidential

after a reset When the OS is ready to configure the system it will reset all of the SAM slots. Each existing SAM will supply a maintenance interrupt.

Polo and HSC2 have 4 PCI slots per PCI bus. There will be one maintenance interrupt per PCIB/ HSC2. Software must poll each PCIB/HSC2 slot for the existence of a PCI adapter. The PCI adapters are addressed as though they are pseudo SAMs (see Section 8.5). Polo needs to check each slot for multi-function adaptors see section 8.5.1.

8.5 PCI Address formation

Gambit/Mirage will format configuration addresses from config_addr as explained in the PCI rev 2.0 spec.

8.5.1 Configuration cycle generation

The hardware will compare the "Bus Number" frcm the address to the bridges secondary bus number which in our system is 0. It will generate a type 0 configuration cycle when the "Bus Number" is zero otherwise it will generate type 1 configuration cycle. When a type 1 configuration cycle is created the lower 2 address bits AD[1 :0] need to be changed from 2'bOO to 2'b01.

For electrical reasons we require that the address for a configuration cycle be Pre-driven. Since the address lines will be resistively coupled to the IDSEL lines going to the adapter the IDSEL lines will have a low slew rate. The address must be stable TBD clocks before FRAME#.

8.5.2 Special cycle generation

When the config_addr register is loaded with the Device and Function numbers of all ones and the Register number is zero the bridge is "Primed" for a special cycle. When our config_data register gets written we generate a special cycle if the bus number equals zero. Otherwise we will send it on as a config_write. Lower level PCI-PCl bridges that match the Bus number will then create special cycles on their secondary buses.

29 December 1995 _ 57

0 781

Polo Software Programming Guide Stratus company Confidential

Figure 24. Gambit/Mirage Bridge Translation config_addr type 00 -> PCI AD

config_addr

31 PCI AD mo 8 2 1 0

Figure 25. Bridge Translation config_addr type 01 -> PCI AD config_addr

31 30 24 23 16 15 11 10 8 7 2 1

31 24 23 15 11 10 8 7 2 1

PCI AD

8.5.3 Polo/HSC2 Configuration Address Generation

Polo/HSC2 have 4 PCI buses with 4 PCI slots each. The devices will be numbered 1-4 on each PCI bus. For type 00 configuration cycles the Gambit will form PCI address bits (31 : 11 ] from the device number in the PCI config_addr register along with bits (16:15] of the Xbus Golfbus IO Space address from the cycle that accesses config_data to mimic which of the 4 pseudo SAMs is being addressed. If the 'Device Number' equals zero the bridge is accessed if it is non zero device 1 is accessed. Therefore device 1 should always written to config.addr to access an adapter. See example below. The IDSEL[3:0] will be tied to AD[15:12] respectively again see section 3.6.4 of the PCI spec for electrical details. Polo/HSC2 also needs to check each slot tor multi-function adaptors see Section 8.1.2.1.

29 December 1995 58

Polo Software Programming Guide Siratus Company Confidential

8.6 Multi-function Adaptors

A PCI adapter can have multiple functions on one card. For example SCSI and enet or multiple SCSI ports. These sub functions are addressed through the function number field in the config_addr register. All cards must use function number 0. If they have multiple functions then bit 7 of the Header Type configuration register indicates whether a device is a multi-function device.

If a multi-function device is detected all possible function numbers must be polled for existence. The PCI spec, allows them to occupy any function number after function 0. Each function has its own configuration header and therefore has to be configured into PCI address space as an individual device.

8.7 SAM and PCI Adaptor Configuration

Once the bridge device is detected the bridge can be used for PCI configuration cycles to detect resident PCI devices without any bridge configuration. Software then reads the header information for PCI device 1. The PCI device needs to be mapped into PCI address space. First the software has to determine how much memory and IO address space the device needs. This is done by writing all ones to the base address registers and reading them back. The device will return O's in all don't care bit positions effectively specifying the address space required.

Note: The typical device will have two base address registers in the header i.e. requiring one memory range and one 10 range. The low order bits distinguish between the memory and 10. See PCI local bus spec section 6.2.5.1.

8.7.1 Base Address Registers

The number of high order bit that an adapter implements determines how big its memory space is. For example Gambit/Mirage responds to 256 MB so it implements the 4 high address bits. If software did not know how big the range was it could write all ones to this register and it would read back f000_0000h indicating the size and the fact that it is a memory space base register. See figure 26 and figure 27 for the formats of base registers.

29 December 1995 59

T/US97/09781

Polo Software Programming Guide Stratus Company Confidential

Figure 26. Base Address Register for Memory

Figure 27. Base Address Register for I/O

8.8 PCI Address Space

Xbus/Golfbus addressing of PCI address space is well defined. Inside each SAMs address space there is a 64KB memory window and a 32 KB I O window from the Xbus/Golfbus. The PCI card's memory may be much larger than these windows and the PCI card needs to mapped into a complete range of PCI address space. Furthermore for Polo we need for these ranges to be non- overlapping from SAM to SAM (remember that there are 4 pseudo SAMs per PCI bus in Polo).

This non-overlapping address space needs to be tracked by the configuration software. There are a number of issues around this PCI address space related to board insertion/removal. When a new board ts added to the system its memory and 10 ranges have to be mapped so that it does not overlap with any previously mapped devices. When a board is pulled it will leave holes in this PCI address space. This bnngs up the need to defragment the PCI address range if new devices cannot be mapped into the existing ranges.

(There are other alternative here like a fixed per slot allocation which may not work due to large possible memory and 10 ranges. The contiguous range has to work per PCI bus. Four separate ranges can be tracked, one for each Polo PCI bus, or one per bucket if that is easier.)

8.9 PCI Bus Target and Master Aborts

The bus master detects a master abort when no target responds to its request. The master sets bit 13 in tiie its PCI configuration status register. The Bridge (Gambit/Mirage) also watches the PCI bus for master aborts and sets a bit in its PCI Error register. This condition will cause a maintenance interrupt it it is not masked off in the SAM 10 space Configuration register. If the

29 December 1995 60

Polo Software F . ^ramming Guide _^ atus Company Confidential

bridge acting as a master detects a master abort it sets bit 13 in the its PCI configuration status register and may cause a maintenance interrupt if it is not masked off in the SAM IO space Configuration register.

A target signals a target abort when it gets an error on a transaction. The target sets bit 11 in its PCI Configuration status register. The master sets bit 12 in its PCI Configuration status register. In Polo and HSC2 the bridge is either the master or target in every transaction. The bridge is aware of the abort and sends a maintenance interrupt again based on the mask bits in the SAM IO space Configuration register.

8.9.1 PCI Bus Fault Detection and Tolerance

See M+D section 9.

8.10 On Line Adds

The PCI configuration will have to be updated when a Jetta HSC2 or a Polo/PCI card is added to the system. This mechanism is different for HSC2 and Polo.

Although the HSC2 board itself can be inserted, PCI cards themselves cannot as they are connected on the HSC2 board via PMC (PCI Mezzanine Card) Adaptors.

Polo requires the PCI bay to be powered down for a board insertion. When a bay comes back on line after a power down the OS should go through a full configuration sequence.

8.11 On Line Failures

An on-line failure is a condition that breaks the SAM. These include SAM errors and PCI errors. See the error registers in section 12.6.

1. Reset the failed device.

2. Check the failed device.

3. Configure it back into the same place it used to be.

8.12 Board removal

Polo requires the PCI bay to be powered down for a board removal.

Although the HSC2 board itself can be removed, PCI cards themselves cannot as they are connected on the HSC2 board via PMC (PCI Mezzanine Card) Adaptors

8.13 Special Requirements

On reads from bit encoded fields with reserved bits software may not rely on the value in these fields. Software must use the appropriate masks. On writes software must read the register and merge in the reserved bits to the new data to be written then write back the merged word.

We need a mechanism for programming and storing config information for PCI cards with user selectable config. options. For example we need to change the SCSI host adapter id number for the second host on a dual initiated bus. There is additional information on this topic in the PCI ECR Status document dated 5/11/94.

29 December 1995 61

Polo Software Programming Guide Stratus company Confidential

When an expansion prom is decode enabled the software must not access the device through any other base address registers. This is because PCI adapters may save gates and share the address decoders for the expansion ROM with another base register.

See the byte order discussion in section 5.6.

8.13.1 Lock requirements

The page 2 SAM registers (section 12.6.2) need to be locked due to Polo multi registers. See section 5.5 for more information on locking these registers.

29 December 1995 62

Polo Software Programming Guide Swatus Company Confidential

9. PCIB Maintenance & Diagnostics

9.1 Overview

The Maintenance and Diagnostics (M&D) subsystem is a major part of the value-added in a Stratus system. It is responsible for managing the configuration of the machine as parts fail and are replaced, and as the machine is upgraded on-line. It handles the reporting and logging of error information, but does not attempt to diagnose the root cause of a system problem by interpreting the error status.

Polo's PCI subsystem consists of 1 to 14 PCI cards arranged in two 7 slot groups. Each group has a controller card (the PCIB) bridging between the Xbus and two multi-slot PCI buses, and a PCMCIA flash card on the PCIB. The HSC2 subsystem consists of at least one (simplexed) HSC2 board with 1 to 4 PCI cards. On Polo and HSC2, the "SAM" is just an abstraction in the address space; 8 "SAMs" are implemented on a single PCIB card, 4 "SAMs" are implemented on a single HSC2 board, and all power up and down simultaneously. The only time new SAMs (PCI cards) appear is when a PCIB/HSC2 is powered up.

9.2 ID Proms

There is a Jetta-style ID prom on the PCIB and HSC2 for tracking board revisions and for error logging. The ID proms on the PCIB and HSC2 should not be programmed with information concerning the PCI card in the SAM's PCI slot

There are no Stratus ID PROMs on the PCI cards themselves; these boards are identified by reading their PCI-defined configuration space. All PCI cards provide a Vendor ID (assigned by the PCI Special Interest Group to assure uniqueness), Device ID, and Revision ID. The PCI cards will not contain individual serial numbers in an on-line readable format. Most PCI cards have bar-code serialization only.

The Polo backplane ID Prom is accessed through the ReCC, identically to Jetta. Additionally, the Polo I/O power supply has 8 bits of ID information in the register space of the gambit ASIC.

9.2.1 Fault Logging

An important function of the ID proms is the logging of fault information to aid in diagnosis when a defective board is returned from the field. Each ID prom records faults in a 512 byte partition. The partition initially starts out empty, and it is filled in a round-robin fashion as faults occur.

Note that PCI cards do not support memory dumping. Any information from a stuck PCI card must be copied prior to resetting the card. Virtually all PCI cards zero memory on reset.

The following registers are logged for each board type. Note that different boards record different registers.

9.2.1.1 PCIB

A single PCIB is a bridge to 8 PCI slots. Accordingly, the relevant per-slot SAM registers appear 8 times in the fault log. In the following table, 'xxx' takes on the hex values 404, 40C, 414, 1 C, 424, 42C434, 43C. The four config space registers are the first 4 DWORDs of the PCI config space in the PCI card in the corresponding SAM; the M&D performing the logging must access them using

29 December 1995 63

Polo Software Programming Guide Stratus company Confidential

the PCI configuration space mechanism. None of the per-bus registers are judged useful for diagnosing failures, and they are not included.

Table 13. PCIB registers logged

9.2.1.2 HSC2

A single HSC2 is a bridge to 4 PCI slots. Accordingly, the relevant per-slot SAM registers appear 4 times in the fault tog. In the following table, 'xxx' takes on the hex values 404, 40C, 414, and 41 C. The four config space registers are the first 4 DWORDs of the PCI config space in the PCI card in the corresponding SAM; the M&D performing the logging must access them using the PCI configuration space mechanism. None of the per-bus registers are judged useful for diagnosing failures, and they are not included.

29 December 1995 64

Polo Software Piu ramming Guide buatus Company Confidential

Table 14. HSC2 registers logged

9.2.2 I/O Power Supply Faults

There are 2 status bits that identify that an I/O Power supply in a Polo system has failed. This information will tentatively be stored in ReCC status registers. Specifically, I/O power supply fault status will be stored in the ReCC bus and Local fault status register (address OxFOOOOB). Bit 2 will be used for I/O power supply 1 fault, and bit 1 will be used for I/O power supply 0 fault.

9.3 Insertion & Removals

9.3.1 PCIB devices

After insertion, power-up and release of reset, a PCIB issues a maintenance interrupt to the host CPU. The host then goes through the process of setting the board on-line: read the id prom and verify revision compatibility with the rest of the system set the yellow led on • TBD

When the lid to one of the Polo PCI card bays is opened, the power to the PCIB and 8 associated SAM slots (7 physical PCI slots and the flash ROM) is switched off. To software, this is identical to

29 December 1995 65 Z 3

Polo Software Programming Guide Stratus company Confidential

a simplexed MIO being pulled from a system. In theory, if all disks are duplicated on different controllers, and if comm is duplicated in both PCI bays as needed, opening a PCI bay lid on a running system should cause no problems. However, it may be advantageous to include an M&D function that quiesses all operation in a PCI bay in preparation for powering it down. From a signalling standpoint, the yellow LED will be illuminated on the remaining power supply

9.3.2 Polo PCI devices

In the Polo and HSC2 systems, there are no SAM wrapped around the PCI cards to provide hot plugging. As a result, new PCI cards appear in the system only when a PCIB or HSC2 powers on, and the code to add a PCI card is simply run after the code to add a PCIB HSC2. In Polo there ts a single MAP ram in the CPU board; in Jetta there are separate MAP rams in each HSC2.

9.4 Reset Overview

All Xbus/Gbus boards support at least two levels of reset, a "warm" reset and a "cold" reset.

Cold reset resets as much state as possible on a board. It is always applied to a board by hardware upon first powering it up so that the C/D sides are sufficiently similar to permit operation of the board without breaking.

Warm reset resets as little state as possible. It is intended to unwedge a broken board with destroying as tittle state as possible so as to permit the diagnosis of fault conditions.

As part of the reset process, hardware first puts a board into the broken state and then about 1 microsecond later unbreaks the board. This has two consequences:

There is an approximately 1 microsecond period during reset that a board will not respond to bus activity.

A board generates a maintenance interrupt when it breaks or unbreaks. This means that a healthy unbroken board will generate two maintenance intenupts as part of reset process and a broken board unbroken by reset will generate a single maintenance interrupt.

This next section will attempt to summanze the effect of reset on the major components of the subsystems. For detailed bit-by-bit descπptions, please refer to Register Definitions on page 82 of this document.

9.4.1 Polo Resets

9.4.1.1 CPU Reset

A Polo CPU can be warm or cold reset through two mechanisms. First, an Xbus reset can be issued to a Polo CPU through an I/O write across the Xbus to the CPU's reset register. This will cause a reset only to a non-broken CPU. Since broken boards do not respond to Xbus accesses, a reset to a broken board will not reset it Polo implements a second reset function designed to reset a board regardless of its state. This is the suggested method for resetting boards in a Polo system, although the Golfbus method is supported for compatibility. This reset is accomplished by performing a local I/O write to the appropriate reset bits in the local Board Reset register. Setting the appropnate bit in this register causes a reset to be performed using a special set of dedicated three way voted lines on the Xbus. For more details on the protocol of this reset line, please refer to the Xbus Functional Specification. This reset does allow for cold and warm resets to be broadcast to any board in a Polo system, including CPUs.

29 December 1995 66

Polo Software Programming Guide tairatus Company Confidential

The effect of a reset on a CPU is identical to a reset on a Golfbus system and is documented in the Golfbus Functional Specification.

9.4.1.2 PCIB

In the PCIB subsystem there is a 2 level hierarchy of resets. The two levels are the one MIO compatible Xbus Board Register, and the eight page 0 SAM compatible Reset registers. Figure 28, below shows a block diagram of the Resets relative to Polo.

29 December 1995 67

Polo Software Programming Guide Stratus company Confidential

Figure 28. Polo PCIB reset Hierarchy

The MIO compatible Xbus/Golfbus Board Register, documented in the Golfbus Functional Specification, allows for either a warm or cold reset to be performed. The effect of this reset is to reset the entire PCIB subsystem. From an FTIO standpoint, this reset is the equivalent of resetting

29 December 1995 68

Polo Software r . ^ramming Guide - .atus Company Confidential

an FTIO and every SAM connected to it. Warm and Cold resets leave the PCIB unbroken, off-line, and with only the red LED on. Warm reset leaves most register state (particularly error register state, PCI configuration space and the PCI boards themselves.) intact; Cold reset clears nearly everything. This reset will reset the MIO compatible Xbus/Golfbus registers, all 8 sets of page 0 per slot registers, both per bus registers, the two PCI buses connected to the PCIBs (cold reset only), all PCI devices connected to the PCIB (cold reset only), both sets of PCI configuration space information (cold reset only), and any associated state with the PCIB. This reset is intended to allow for resetting of a broken PCIB. As with the CPU resets, this can be accomplished either across the XBus or from a local CPU write. As with the CPU, the local write is the preferred method.

The page 0 SAM compatible Board Reset Register is supported to allow for resetting an individual SAM. A reset to this register leaves the PCI slot in the Off-line not ready state.This reset resets the pagO SAM state for that slot, the PCI board in that slot (cold reset only), and any state associated with that slot. This reset is intended to allow software to revive a dead PCI card, recover from a single PCI card hang, and provide any compatibilities required between an FTIO system and a Polo system with regard to reset functionality.

9.4.1.3 PCI cards

PCI cards are provided by third party vendors, and do not support the warm and cold flavors of reset. All PCI cards implement a specified reset behavior for their PCI interface; the effects of reset on the remainder of the card is card-specific.

Typically, resetting a PCI card clears all useful registers and zeros out on-board memory. Please refer to the manufacturers specification for each PCI card.

This reset is accomplished by writing a PCI reset to the appropriate page 0 SAM compatible board reset register. It has the effect of only resetting the PCI card and not effecting an SAM compatible or PCIB board state.

29 December 1995 69 s | l

Polo Software Programming Guide Stratum Jompany Confidential

9.4.2 HSC2 Resets

9.4.2.1 HSC2 tn the HSC2 subsystem there is a 2 level hierarchy or resets. The two levels are the one MIO compatible Xbus Board Register, and the four page 0 SAM compatible Reset registers. Figure 28, below shows a block diagram of the Resets relative to HSC2.

Figure 29. HSC2 PCIB reset Hierarchy

The MIO compatible Golfbus Board Register, documented in the Golfbus Functional Specification, allows for either a warm or cold reset to be performed. The effect of this reset is to reset the entire HSC2 subsystem. From an FTIO standpoint, this reset is the equivalent of resetting an FTIO and every SAM connected to it. Warm and Cold resets leave the HSC2 unbroken, off-line, and with only the red LED on. Warm reset leaves most register state (particularly error register state, PCI configuration space and the PCI boards themselves.) intact; Cold reset clears nearly everything. This reset will reset the MIO compatible Golfbus registers, all 4 sets of page 0 per slot registers, both per bus registers, the PCI bus connected to the HSC2 (cold reset only), all PCI devices connected to the HSC2 (cold reset only), the PCI configuration space information (cold reset only), and any associated state with the HSC2. This reset is intended to allow for resetting of a broken HSC2.

The page 0 SAM compatible Board Reset Register is supported to allow for resetting an individual SAM. A reset to this register leaves the PCI slot in the Off-line not ready state.This reset resets the pago SAM state for that slot, the PCI board in that slot (cold reset only), and any state associated with that slot. This reset is intended to allow software to revive a dead PCI card, recover from a

29 December 1995 70 *

Polo Software Programming Guide Stratus Company Confidential

single PCI card hang, and provide any compatibilities required between an FTIO system and a Polo system with regard to reset functionality.

9.4.2.2 PCI cards

PCI cards are provided by third party vendors, and do not support the warm and cold flavors of reset. All PCI cards implement a specified reset behavior for their PCI interface; the effects of reset on the remainder of the card is card-specific.

Typically, resetting a PCI card clears all useful registers and zeros out on-board memory. Please refer to the manufacturers specification for each PCI card.

This reset is accomplished by writing a PCI reset to the appropriate page 0 SAM compatible board reset register. It has the effect of only resetting the PCI card and not effecting an SAM compatible or HSC2 board state.

29 December 1995 71

^Λ 2, 9

Polo Software Programming Guide Stratus Company Confidential

9.5 Determining the source of a Maintenance Interrupt

Maintenance interrupts provide notice that there one or more conditions requiring action from the M&D process. In general, it is impossible for software to distinguish between a single maintenance interrupt, and multiple maintenance interrupts in quick succession. Upon seeing a maintenance interrupt in a system, the following process can be used to determine the source(s) of the interrupt.

If an Golfbus/Xbus board that was in the system is no longer visible to software then that board generated a maintenance interrupt. (The recommended way of determining if an Xbus board is unbroken is by reading the Bus Interface Status Register; it is guaranteed to return non-zero on a unbroken board). To get more detail on what happened to the board, it can be reset and its Common Broken Status and ASIC Specific Broken Status registers examined. If multiple reset attempts have no effect, the board was either removed or is too broken to be read. The OS should then use the board presence information from the RECC to determine if a board has been removed.

Golfbus/Xbus boards that have not broken or been removed from the system will have either the "Bus Interface Maintenance Interrupt" bit or the "Board Logic Maintenance Interrupt" bits set in the their Bus Interface Reporting register if they activated maintenance interrupt.

If the "Bus Interface Maintenance Interrupt" bit is set, then Board Reset register may be read to see if a maintenance interrupt was due to a board reset. Similarly, the Bus Info Error Status and Misc. Error status registers may be read to determine if the maintenance interrupt was generated by any non-fatal errors (the board would be broken otherwise). If no bits are set in either of these two registers, the maintenance interrupt was software generated.

If a "Board Logic Maintenance Interrupt' is set on a PCIB or HSC2, then the interrupt originated from SAM slots associated witii this board, or from a disk being inserted or removed from the Storageworks disk shelf. The next step is to read the SAM Maintenance Attention Request register. If any of bits 3-0 in SAM Maintenance Attention Request register is set, all tour PCI Bus Interface State register for SAM3 - SAMO should be read. If any of bits 11 - 8 in SAM Maintenance Attention Request register is set, all four PCI Bus Interface State register for SAM7 - SAM4 should be read (Polo only).

In Polo, if a "Board Logic Maintenance Interrupt' is set on a Polo CPU, then the interrupt originated from an IOVA map error and the IOVA Map Error 1 and 2 registers should be examined (section 12.4.3 and section 12.4.2). The TRID can be used to determine the PCI slot that sourced the Xbus IOVA operation. The IOVA map error maintenance interrupt can also be masked from the IOVA Map Error 1 register.

The next step is to examine the SAM Status and Configuration registers for all PCI slots that issued maintenance interrupts. Bits 7:21 specify various sources of maintenance interrupts; only some bits are implemented in Polo & HSC2 systems.

On Polo systems, the final place to check is the Polo Disk Status register. This indicates whether a maintenance interrupt was issued due to a disk swap event. Unfortunately, it does not specify whether a disk was added or removed; only that there was a change of state.

9.6 Faults & Errors

9.6.1 PCI card faults

Typically, there will be no Stratus style fault detection comparison logic on 3rd party PCI cards. Some boards may compute check-sums or utilize parity on memory arrays; support for this level of fault reporting is the responsibility of the card driver. However, there should be an interface for drivers to communicate a text string of fault information to the M&D, so it can be incorporated into

29 December 1995 72

Polo Software Programming Guide Stratus Company Confidential

the SYSERR log and be available to the CAC.

The presence or absence of a PCI card is indicated in the PCI config space Vendor ID register. The PCI bridge will return a value of all 1 s (an invalid vendor ID) on read accesses to configuration space registers of non-existent devices.

9.6.2 PCI bus faults

9.6.2.1 PERR#

The PCI bus is protected by a single Even parity bit over 36 bits of information (32 address data and 4 C/BE). The parity is driven with the same timing as the address/data, but delayed by one clock. All agents on the PCI bus must generate parity.

Data parity errors are indicated using the PERR# signal. Only the device which claims the cycle may drive the PERR# signal on the PCI bus. The Gambit/Mirage ASIC will continuously monitor PCI transactions and the state of the PERR# signal, regardless of the selected device. Any PCI parity errors observed during non-Special Cycles will result in a maintenance interrupt action back to the host CPU.

PERR# is a non-recoverable condition and causes a state reduction in the Gambit/Mirage ASIC from on-line to off-line, not ready. This reduction in state will cause a maintenance interrupt It is up to software to test and bring the card back into service if it can.

9.6.2.2 SERR#

System errors which could be catastrophic are indicated on the SERR# signal. Any PCI agent which detects a non-recoverable condition can assert SERR#. The Gambit/Mirage ASIC monitors the state of the SERR# signal and will issue a maintenance interrupt up to the host processor for any assertion of SERR#. The state of the SAM will then be reduced to Off-line, not ready. The Gambit/Mirage ASIC does not generate target-abort for this case.

A PCI adaptor non-recoverable condition includes:

1. Address parity errors on all cycles.

2. Data parity errors on Special Cycles.

3. A catastrophic adapter fault.

SERR# can be disabled via config space on all PCI devices (including the Gambit asics). All PCI devices power-up with SERR# disabled; it is up to M&D software to enable SERR# assertion.

9.6.2.3 Polo(Gamblt)/HSC2(Mirage) Error logic

Polo(Gambit) detects several error conditions on PCI bus and takes different actions depending upon error kind. Actions taken for different kind of error is given below.

1. Target Abort

If the bridge is current bus master, bring the SAM who drove the devsel line to OFFLINE-NOT- READY state, if this is a read access by host, return failed op.

If the SAM is current bus master, bring the current bus master to OFFLINE-NOT-READY

29 December 1995 73

Polo Software Programming Guide Stratus υwmpany Confidential

state.

Set the value in PCI Error register and PCI IOVA Error register.

2. Master Abort

If the bridge is current bus master, and this is a read access by host, return failed op.

If the SAM is current bus master, bring the current bus master to OFFLINE-NOT-READY state.

Set the value in PCI Error register and PCI IOVA Error register.

3. Parity Error

If the bridge is current bus master, bring the SAM who drove the devsel line to OFFLI E-NOT- READY state.

If the SAM is current bus master, bring the current bus master to OFFLINE-NOT-READY state.

Set the value in PCI Error register and PCI IOVA Error register.

If the bridge is bus master and is reading from the SAM, failed op returned to the host.

If the bridge is bus master and is writing to SAM, it is up to the SAM to take appropriate actions.

If the SAM is current bus master and is writing to the host, the transactions is dropped. No transaction happens on X bus.

If the SAM is current bus master and is reading from the host, it is up to the SAM take appropriate actions.

4. System Error

If the dont_break_on_serr bit is not set in test control register, bring all SAMs to OFFLINE- NOT_READY state. If the dont_break_on_serr bit set in test control register, nothing happens. On power on, dont_break_on_serr bit is cleared.

Set the value in PCI Error register and PCI IOVA Error register.

If the bridge is bus master and is reading from SAM, failed op returned to the host.

If the bridge is bus master and is writing to SAM, it is up to the SAM to take appropriate actions.

If the SAM is current bus master and is writing to the host, the transactions is dropped. No transaction happens on X bus.

If SAM is current bus master and reading from the host memory, the transaction is dropped. No transaction happens on X bus. The SAM receives target abort

Note: If the host read of PCI bus gets SERR and retry, but do not get SERR for retried Error register is

the devsel line. If

29 December 1995 74

Polo Software Programming Guide Stratus Company Confidential

6. Drive Check Error

Break PCIB/SAM board.

Set the value in PCI Error register and PCI IOVA Error register.

Set the value in PCI Error register and PCI IOVA Error register.

9. Peer to Peer Error

Bring the current bus master to OFFLINE-NOT-READY state. Set the value in PCI Error register and PCI IOVA Error register.

10. Protocol Error

If the bridge is current bus master, issue PCI reset to the SAM who drove the devsel line. If this is a read access by host, return failed op.

If the SAM is current bus master, issue PCI reset to the SAM.

Set the value in PCI Error register and PCI IOVA Error register.

Note: If a PCIB is issued a warm reset while PCI traffic is in progress, a protocol error may be detected by Gambit error logic.

In certain cases of protocol error, write buffer data in ti e Gambit ASIC goes on X bus even if there is a protocol error. ( e.g. A pci adapter does a write to gambit, but does not assert irdy_ and deasserts trame_, causing a protocol error) .

11. Host Request FIFO Time-out Error

This error is detected when Host Request FIFO (inside Gambit ASIC) requesting to access PCI bus fails to advance by the number of clock ticks specified in Host Request FIFO Time-out Value Register. In this case, PCIB broken so that CPU does not hang, waiting for data to return from PCIB.

Break PCIB/SAM board.

Set the value in PCI Error register and PCt IOVA Error register.

Note: This error happens in some rare catastrophic condition(e.g. frame_ signal stuck low on PCI bus). When this error condition happens, the PCI IOVA Error Register and PCI Error Register may or may contain the info about the transaction that caused this error, since it Is impossible to figure out which transaction caused this error.

The PCI error logic in Gambit implements PCI IOVA register and PCI Error register for diagnosing errors. The PCI Error Register is read only by host. This register has arm rearm control. This register could be armed/rearmed by host. When PCI Error Register is armed, the PCI IOVA

29 December 1995 75

~ V

Polo Software Programming Guide Stratus company Confidential

register contains the PCI bus address that caused error. When an error occurs, the PCI IOVA register and PCI Error registers are frozen so that it holds the address that caused error. Subsequent errors does not affect the register. It needs to be rearmed before it could store address for next error. For description of PCI IOVA register and PCI Error register, refer to section 13.5.5.

9.6.3 Disk bus faults

The DEC Storageworks disk solution provides the necessary features for Stratus disk M&D. In particular, Storageworks accomplishes live insertion, insertion/removal detection, and fault LEDs. In order to perform these tasks DEC has implemented two basic features. The first one deals with live insertion. Live insertion is accomplished through some current limiting and power sequencing. It seems to work. The latter two features are accomplished through DEC'S fault bus. The fault bus consists of a swap interrupt and an LED bus. Both of these features are implemented on Polo.

There are a series of operations which should be performed by the M&D processes associated with the various M&D features of the disk subsystem. These are discussed in the sections below.

9.6.3.1 LED Initialization

LED initialization should be performed at power up time. The LED initialization routine provides a simple way to turn off all of the LEDs. At power up time, the LED state is non-deterministic, and must be set by setting/resetting each LED individually, or as a group by using the broadcast bit. The broad cast is performed by setting (turn on LEDs) or clearing (to turn off LEDs) bit 4 and setting bit 3 (the broadcast bit) in Disk LED Control register. This register is described in section 12.6.1.1 of this document. In a broadcast write to the disk LEDs, the SCSI device target ID (bits 2:0) are not meaningful. For example, to set all of the LEDs in a SCSI bus, a 18x would be written to the register.

9.6.3.2 Disk LED Control

Individual LED control is performed on an as needed basis by the software. Software has control over one of the LEDs, the fault indicator, on the disk. In order to address a particular slot, bit 3 (the broadcast bit) must be clear, bits 2:0 should be set to the SCSI device target ID, and bit 4 should be set (to turn the LEDs on) or cleared (to turn the LEDs off) in the Disk LED Control register. This register is described in section 12.6.1.1 of this document. The SCSI device target ID is the same as the SCSI devices normal SCSI device number. For example, to set the led for SCSI device 2, a 012x would be written to the Register, to clear tiie LED for the sam device, a 002x would be written to the register.

9.6.3.3 Disk Insertion and Removal

Finally, disk insertion or removal is detected through the swap has occurred bits. For the C-side shelf, this is on bits 1 and 17 and for the D-side it is on bits 9 and 25 of the Disk Status register. If the corresponding interrupt disable bit is cleared, then a maintenance interrupt will be generated due to this event. The interrupt is cleared by clearing the appropriate bit in the register, bit 1 for C- side interrupts, and bit 9 for D-side interrupts. The swap interrupt reflects only that a change has occurred in the state of the disk shelf, the M&D software must perform a SCSI poll to determine the nature of the change.

29 December 1995 76

Polo Software Programming Guide Stratus Company Confidential

9.6.4 SCSI bus faults

Faults on the SCSI bus are handled in a manner specific to the PCI card.

9.6.5 PCIB/HSC2 faults

9.6.6 Xbus/Golfbus faults

Xbus faults reported by the PCIB/HSC2 are handled like those from any other Xbus Golfbus board. Please refer to the Xbus/Golfbus Functional Specification for more information.

9.7 Diagnostics

9.7.1 PCI Card

The level of diagnostic support for the PCI cards will be consistent with the PCI Third-Party Adapter Requirements Specification. Most PCI cards will be very highly integrated (consisting of 1 or a few chips), and all are anticipated to be very reliable. In any case, they are non self-checking components, and the system is protected from incorrect operation by higher level mechanisms: address re-mapping, check-sums, and time-outs.

The system must be able to isolate faults down to a field replaceable unit. If it is not possible to isolate faults to this level via the error reporting registers, specific fault isolation diagnostics will be needed.

9.7.2 PCIB

The PCIB/HSC2 board has no local processor and an extremely low chip count. It will be tested and diagnosed exclusively by scan.

29 December 1995 77

Polo Software Programming Guide Stratus ompany Confidential

10. ReCC Functionality on Polo

This section describes key changes envisioned for the ReCC firmware on Polo. On Polo, it is a goal to isolate the OS from many changes in the M&D subsystem that are particular to Polo.

10.1 Remote Power control

One feature on Jetta that is not supported on Polo is that ability to cycle power from the ReCC. The Polo machine does not implement house keeping power or the power supply control necessary to perform this function. Therefore it is not supported on Polo. ReCC firmware should probably stub out this command set. It has been suggested that the ability to power down the machine from the ReCC be supported. This feature would require spinning the else gate array on the ReCC and therefor will not be supported.

10.2 RS-485 us

Polo does not implement the CDC. In the initial release of Polo there is no 485 bus devices. The 485 is in the system for future expansion but with no devices in the 485 system, it will not respond to any polls. Therefore, the ReCC firmware should return intelligent response to M&D requests for 485 information.

10.3 Powerfail

Polo does not support powerfail. Polo's solution to AC failures is accomplished through an interconnect to an external UPS. The interconnection to the UPS will be documented in a later release of this specification. It is expected to be RS-232, and it will be connected to the secondary console when it is used.

10.4 Clockcard, backpanel power supply failures

The clock card and backpanel power supplies are not implemented in Polo, therefore there will be no signalling of these faults.

10.5 I/O power supply failures

The I/O power supply fault signals may be rooted in place of the backpanel power supply signals. If these lines are activated it signals that the I/O power supply has determined that a fault exists in the I/O power supply.

10.6 NVRAM

It would be preferable to replace the full functional EEPROM with flash prom. This may effect the writing algorithm tor the PROM.

29 December 1995 78

Polo Software Programming Guide Stratus Company Confidential

11. Board States

This section describes the board states for the PCIB and SAM modules in the Polo system and the HSC2 for the Jetta system.

11.1 PCIB Board States

The Polo PCIB module would gain little by supporting duplexed operation for the following reasons:

1. All paired SAM space accesses, whether to memory or I/O must be converted in hardware to non-paired. This is because the Polo PCIB does not have the benefit of going through an MIO before driving the Xbus so the two PCIB boards cannot know the same information.

2. Many of the registers in Gambit could only be accessed through paired space due their definitions already.

The task of implementing a duplexable PCIB in hardware would add work with little or no gain so the PCIB can only be run in simplexed mode.

11.2 HSC2 Board States

The HSC2 is simitar to the BIO in that it does not support duplexed operation. The HSC2 is similar to the Polo PCIB in that it shares the SAM board states model detailed in SAM Board States on page 80.

26 December 1995 79

Polo Software Programming Guide Stratus Company Confidential

11.3 SAM Board States

Figure 30. SAM-PCI Board State Transition

Board States:

OFF-LINE - Not Ready

Board capable of receiving and responding to XBus/Golfbus requests, but is incapa¬ ble of performing PCI adaptor initiated requests. There is non-zero error state in the SAM.

This state indicates the occurrence of a. PCI Bus related hardware fault which was detected by the checking logic. This state cannot be entered by a software set off-line command as there must be non-zero error state in the SAM.

OFF-LINE - Ready

Board capable of receiving and responding to XBus/Golfbus requests but is incapa¬ ble of performing PCI adaptor initiated requests. No error conditions are present.

This state is entered automatically after a cold or warm reset (provided there are no errors detected by the SAM) or when the error state of the chip is cleared via a Re¬ arm Error Command. Also software can enter this state by either a set off-line com¬ mand or by a clear on-line command.

ONLINE

Board capable of receiving and responding to XBus/Golfbus requests. Board capable of performing XBus/Golfbus initiated PCI requests. PCI adapter enabled to post re¬ quests.

26 December 1995 _n 80

Polo Software hiogramming Guide otratus Company Confidential

Software can enter this state with a set on-line command to the Bus Interface Sate Register provided there is a non-zero error state.

Figure 31. SAM board states.

KEY:

1 — State bit true. 0 — State bit false. X — State bit true or false.

26 December 1995 81

^ %

Polo Software Programming Guide Strati.- Company Confidential

12. Register Definitions

The following section outlines the I/O space registers in the Polo system.

12.1 XBus/Golfbus Bus Interface Registers

All IO registers in the Xbus Golfbus interface are designed to be accessed as 32-bit registers so as to be accessible by all devices on the bus. Writes to any smaller data size Xbus/Golfbus registers are ignored and reads return 32-bits even if a smaller size datum was accessed. Though all registers are 32 bits, they are spaced at 64 bit (8 byte) intervals to aid future board designs which may make use of 64-bit internal busses.

Writing any non-existent IO register has no effect (i.e. unused addresses do not wrap around and map to used addresses). Reading a non-existent IO register either returns O's or causes a NACK, depending on the particular register. The read never has any side effects.

There are two types of IO registers supported in the Xbus interface in additional to interrupt registers intended to mimic the HP-PA interrupt scheme:

1) readwrite registers

These registers are readable/writable 32 bit registers for which no bit encoding occurs, i.e. assuming hardware events have not altered any state, what is written to bit 0 is read from bit 0, what is written to bit 1 is read from bit 1 , etc. For upward compatibility, unused bits should always be written with O's, though 1 's have no effect. Reads of unused bits always return O's.

2) read set clear registers

Read/setclear registers are used when individual bits need to be atomically updated. Reads return 32 bits; writes specify one of the 32 bits to be either set or cleared. Specifically, writes are performed by selecting one of the 32 read bits bit to be atomically set or cleared with the five least significant bits (4:0) pointing to the bit and using the 8th least significant bit (7) to set (=1) or clear (=0) the selected bit For example, to set bit 2 of the read register, 00000082x is written to the register. To clear bit 2 of the read register, 00000002x is written.

Unused bits always return O's and for upward compatibility unused bits should always be written with O's, although writing 1's has no effect.

3) read clear registers

Read/clear registers are used when individual bits need to be atomically updated. Reads return 32-bits; writes specify one of the 32 bits to be cleared. Specifically, writes are performed by selecting one of the 32 read bits bit to be atomically cleared with the five least significant bits (4:0) pointing to the bit and using the 8th least significant bit (7) clear (-0) the selected bit. For example, to clear bit 2 of the read register, 00000002x is written. Attempts to set the register will have no effect.

Unused bits always return O's and for upward compatibility unused bits should always be written with O's, although writing 1 's has no effect.

12.2 Reading Registers with Different C/D Information (Polo Only)

Note: This section does not apply to the HSC2 as there is no board processor to read the enor registers in local space.

There are certain C and D registers that can have different values (for example the error registers). In a duplexed pair of boards the register values between board pair may also vary. There must be a way of reading these registers via either the local or a remote processor.

29 December 1995 82

Polo Software Programming Guide btratus Company Confidential

These registers should not be read in local space because this could bnng duplexed boards and/ or the C&D board halves out of lockstep when the register values are different.

The access to these registers is similar to unpaired reads in that part of the data checking has to be ignored. During an unpaired read the board not providing the data has to ignore the checking but go through the motions to remain in lockstep. in this case the concept is taken a little further. Each board half (except the board half actually being read) goes through the motions but ignores the error checking on the info; however the trid, func.op, and parity are still checked.

Since some of the bits are driven from the C-side and some are driven from the D-side and the data from the two sides can differ, there is a complication in generating good parity. The solution to this problem is to place the differing error bits in the regions of the info bus that are driven by the ASIC responding to the read (see figure 7). For example C-side error bits have to be in the region 7:0 or 23:16, and the D-side from 15:8 and 31:24. To get good parity each side drives the same data on each of the bytes that the ASIC drives and puts zeros into the bytes that the ASIC does not drive. This will result in an even number of ones in the parity chain.

These registers must be readable when the board is off-line which is not an issue because the board responds to unpaired reads when it is off-line. There is a problem reading these registers if the board is broken. When the board is broken, it cannot be read or written by software. To access these registers on a broken board there must be a warm reset which brings the board to the off¬ line state.

Figure 32. Bus Driver Definition

trid nιnc_op Info Bus Parity

Bits arc assigned starting from the least significant bit.

IK

Driven by the D-side ASIC during normal accesses, and D Driven by the C-side ASIC during normal accesses, and by the driving ASIC during by the driving ASIC during single- side single-side accesses.

Parity is driven by the D-side on the CPU.

It is driven by the D-side of the PCIB during normal accesses, and by the driving ASIC during Single Side accesses.

12.3 Common Xbus/Golfbus Register Definitions (Polo Only)

Note: HSC2 supports the set of common Golfbus Registers. Refer to the Golfbus Specification for details concerning the common Golfbus Registers for the HSC2 board.

The Xbus supports a set of common registers that are present on all Xbus boards. These are modeled directly after the Golfbus interface registers, and preserve compatibility to the bit level whenever possible. However, due to the different physical partitioning of the Polo system, there are necessarily some changes. To ease the task of porting software to the Polo system, we define four levels of compatibility for each register: exact: this register preserves exact read/write functionality with the Golfbus definition, and may be programmed in exactly the same way. Examples include the ID PROM Address Data

29 December 1995 83

A \

Polo Software Programming Guide Stratum Company Confidential

register.

• usage change: this register is bit for bit identical to the corresponding Golfbus register, but the usage in practice may be different. For example: the performance monitor registers can be programmed to monitor the function codes on the backplane; however, the Xbus is 32 bits and does not use the Golfbus 64 bit backplane function codes.

• subset/super-set: the Xbus version of this register is a subset or super-set of the Golfbus register. It may be programmed in the same way, but bit values that make sense in Golfbus systems may not be supported in Xbus systems or visa-versa. It may also have additional bits that do not exist in a Golfbus board. An example is the Bus Interface State register: the Xbus does not have the equivalent of the bus obey, so setting the "obey_a" bit has no effect

• incompatible: this register has new meanings and bit definitions in an Xbus system.

The Xbus/Golfbus compatibility column in table 15 indicates the level of compatibility with the corresponding Golfbus register. Descriptions of the registers follow the table

Table 15. I/O Space Map.

29 December 1995 84

General Warning: Many of these registers have bits which can be toggled during self-test and left in varying legitimate states. Prior to or as part of duplexing, all bits must be ensured consistent.

Second General Warning: Several of the error reporting registers are "CD different' and can legitimately contain data that is different between C and D sides of the board. They must only be read via unpaired space, or they could break the board or hang the system

The format of each register descπption is as follows. The paragraph header for each descnption provides the register name. The bold faced text immediately following contains the register type, address(es), reset states, and a listing of boards types it is present on.

12.3.1 ID Prom lnstr

Type: Read/Write Offset 122:0]: 7FFFF8

Present on all Xbus boards

Warm and Cold have no effect on this register

Compatibility: exact

Writing to this register scans an instruction into the external JTAG ID Prom logic. Reading this register returns the status of the scan. It is only used tor initializing the ID Prom logic after power- up or reset

31:6 Reserved

7:0 JTAG instruction/status for the external ABT18245 scan register. Refer to the next section for usage. Valid instructions are

SAMPLE_PRE OAD ^« = Θ2x EXTEST = OOx

Vatid return status

OK STATϋS 81x

29 December 1995 85

J\ ^ \

Polo Software Programming Guide Stratus Company Confidential

12.3.2 ID PROM Address Data

Type: Read/Write Offset [22:0]: 7FFFF0

Present on all Xbus boards

Warm and Cold have no effect on this register

Compatibility: exact

31:28 ID prom opcode. This controls the WE_, CE_, and OE_ lines of the ID Prom id_READY - 6 id_PRESENT_DATA - Ex id_WRITE = Ax id_OFF - 7x id_READ - x

27:20 ID Prom Data. On writes, this is the data to be written into the id prom. On reads, it is the data returned

19:11 Reserved

10:0 ID Prom Address.

Every Xbus board has a 2Kx8 writable ID PROM. The PROM is intended to store various board status information such as serial number, eco rev level, PCB rev level, etc.

The actual PROM used is the X2816C 2Kx8 EEPROM from Xicor. This part is accessed though a JTAG compatible serial scan interface to save pins on the bus ASIC and to allow the part to be accessible via the board's scan test logic. A Texas Instruments ABT18245 scannabte buffer is used to translate from the scan interface to the ID Proms broadside pins. More detail on the use of this part is available in section 11.3.1 on page 98 (ID PROM Partitioning) in the Xbus Functional Specification.

Logically, the ID PROM appears as two virtual 32-bit registers, an instruction register for sending initialization commands to the ABT18245 scan chip, and an address_data register for reading and writing address/data pairs to the id prom itself. These two virtual registers are actually implemented as a single 44 bit scan register in the ASIC. Writing to the instruction register initiates an 8 bit instruction scan to the ABT18245, and writing to the addressjdata register initiates a 44 bit data scan which drives and samples the pins of the id prom. Reading the address_data register returns the bits of the scan register corresponding to the address, data, and opcode fields. Reading the instruction register returns an 8 bit value corresponding to the results of the instruction scan. Note that there is only one scan register; a read of id prom instruction reg should only follow a write to the id prom instruction, and a read of the id prom address data reg should only follow a write to that address. Otherwise, deterministic gibberish will be returned. Once initialized, reads of the instruction and addressjdata registers will always return non-zero data; if a zero is ever returned, it is due to the target board going broken or a simplexed fault in the ID prom hardware.

The ID Prom hardware needs to be initialized after power-up or a warm or cold reset. Once it has been initialized, the prom can be read or written. The ID prom is a fully random access device, with no sectors or erase modes. When a single byte is written to the X2816C, the device begins an internal programming cycle that takes ~5mS. While the programming cycle is in progress, bit[7] of the byte written to the part will appear inverted when read; software needs to poll the device to determine when the program cycle is finished. The ID prom has an endurance of 10,000 writes.

To initialize the ID Prom hardware:

ID_prom_instr ^■ = SAMPLE_PRELOAD; //(0000_0082x) ID_prom_instr = EXTEST; //(0000_0000x)

29 December 1995 86

Polo Software Programming Guide Stratus Company Confidential

if(ID_prom_instr != OK_STATOS) ; //(0000_00θlx)

"hardware failure in ID prom" ID_prom_address_data = {id_OFF,28'h0000000)

To read the ID prom addresses start_addr to end_addr

ID_prom_address_data «= {id_READ,8'hOO,start_addr} for (i - start_addr; i<= end_addr; i++)

{

ID_prom_address_data = {id_READ, 8'hOO, i+1} read_data[i] «= ID_prom_address_data[27:20]

} ID_prom_address_data - {id_OFF, 28'h0000000]

To write my byte to the ID prom address start.addr

ID_prom_address_data ^β {id_PRESEN _DATA,mybyte,start_addr} ID_prom_address_data = {id_WRITE,mybyte,start_addr} ID_prom_address_data = (id_PRESENT_DATA,mybyte,start_addr)

/* poll to wait for programming cycle to complete */ ID_prom_address_data = {id_READ, 8'hOO,start_addr} /* once */ ID_prom_address_data » {id_READ, 8'hOO,start_addr} /* again */ isbyte= ID_prom_address_dat [27:20] while (isbyte[7] != mybyte[7])

{ wait 1 ms

ID_prom_address_data - {id_READ,8'hO0,start_addr} isbyte- ID_prom_address_data[ 7:20}

1 ID_prom_address_data = {id_OFF, 28'h0000000)

Note that two successive writes of the id_READ opcode are required before read data can be examined. This is because the boundary scan ring in the ABT18245 acquires the data on its pins before applying the new data. The first write acquires old data, then applies the id_READ opcode and address to the ID Prom. The second write acquires the results of tiie read opcode and first address, then applies the id_READ opcode (and optionally a new address).

Note that ID PROMs differ in contents from board to board. Boards operating in lock step should always read and write them in an unpaired fashion. An unpaired read to an ID PROM belonging to one of a duplex board pair causes both board's state machines to perform the actual read, though only one board actually drives the bus with the read data. An unpaired write causes both board's state machines to go through the motions, though TCK is inhibited on the board not being written.

This JTAG scan occurs at 12 MHz, hence each write of these registers takes about 5 usec. The bus ASIC circuitry automatically delays a subsequent access if an earlier initiated write is still in progress. When the ID Prom is accessed via global, paired or non-paired space, other IO accesses can take place while an ID prom access is in progress. When the ID prom is accessed via Local space, all other IO space access to Bus Interface registers are held off until the access completes.

29 December 1995 87 k ς

Polo Software Programming Guide Stratus company Confidential

Figure 33. Internal Implementation of ASIC scan register

Warning: Because multiple processors/processes simultaneously accessing the ID PROM may interfere with each other, software must manage access to the ID PROM to ensure only a single processor/process is accessing it at a time.

Warning: The ID PROM is a simplexed (non error protected) component A checksum or other similar protection scheme should be utilized by software to ensure the integrity of the data read from the ID PROM. Also, because ID PROMs on partnered boards will have different contents, ID PROM Data registers should always be read in an non-paired fashion.

12.3.3 Board Reset

Type: Read Set Clear Offset [22:0]: 7FFFE8

Present on all Xbus boards

Cold and warm reset affect this register as documented below

Compatibility: super-set

Polo boards have two resets, a cold reset applied to the board at power up (or via software or ReCC) and a warm reset applied via software or the ReCC once the machine is running. Cold reset brings all registers to a known deterministic state, warm reset generally resets state machines and cancels operations in progress but leaves as much of the register state untouched as possible to aid in debug.

The activation of any of the bits in this register (excluding test cases) mean that a reset pulse was earlier driven on the board, not that a reset line is currently asserted. It should be noted that as part of the normal reset process, the board goes broken for the duration of the reset pulse, and then unbroken when the reset de-asserts.

29 December 1995 88

Polo Software Programming Guide Stratus Company Confidential

Cold resets clear all bits in this register except for the bit identifying the source of the cold reset, warm resets leave bits in their current state (except for activating ttie bit that identifies the source of the warm reset).

This is a Read/Set/Clear register, commands are performed by writing to this register with data bits [4:0] pointing to a bit to set or clear and bit 7 determining whether the bit is set (=1 ) or cleared (=0). Reading this register returns 32-bits. Setting either "Software initiated Cold Reset' or "Software initiated Warm Reset' actually causes a reset, even if the bit was already set. Setting (to test the register read path) the other bits cause the bits to go to 1 , but no reset takes place.

Polo is different from Golfbus systems in that writing the Board Reset register of a broken board across the bus has no effect For this reason, Polo has 4 additional reset bits that when activated send a reset pulse to the corresponding slot. This reset is inteφreted as a cold reset if it is three 4 Mhz periods long and as a warm reset if the pulse is two 4Mhz pulses long. Software Initiated Cold Reset and Software Initiated Warm Reset still work as before when performed locally, and work across the bus when the target board is unbroken.

31:13 These read bits/write encodings are reserved for additional board specific resets.

12 Cold Reset Slot 3

Writing to set one of these bits sends a cold reset pulse to the board in the corresponding slot. These bits return 0 when read. These bits are only implemented in the CPU.

11 Cold Reset Slot 2

10 Cold Reset Slot 1

9 Cold Reset Slot 0

8 Warm Reset Slot 3

Writing to set one of these bits sends a warm reset pulse to the board in the corresponding slot. These bits return 0 when read. These bits are only implemented in the CPU.

7 Warm Reset Slot 2

6 Warm Reset Slot 1

5 Warm Reset Slot 0

4 Software initiated Warm Reset

When reading this register, this bit ~1 indicates that the board has received a software generated warm reset since this bit was last cleared. Writing a 00000084x to this reg¬ ister generates a warm reset pulse on the board and sets this bit. Writing 00000004x to this register clears this bit. The activation of either of the cold reset bits also clears this bit

3 Reset line initiated Warm Reset

When reading this register, bit 3 =1 indicates that the board has received a warm reset from the reset line since this bit was last cleared. This bit is cleared by writing 00000003x to this register. The activation of either of the cold reset bits also clears this bit This bit can also be set by writing 00000003x to this register.

2 Reset line initiated Cold Reset

When reading this register, bit 2 =1 indicates that the board has received a reset line initi¬ ated reset when the board was broken and therefore treated it as a cold reset Writing 00000002x to this register clears this bit, as does a power up initiated or software ini¬ tiated cold reset. When this bit is set by a reset line reset all other bits in this register are cleared. This bit can also be set by writing 00000002x to this register.

29 December 1995 89

T/US97/09781

Polo Software Programming Guide Stratus company Confidential

1 Software initiated Cold Reset

When reading this register, bit 1 =1 indicates that the board has received a software gen¬ erated cold reset since this bit was last cleared. Writing a 00000081 x to this register generates a cold reset pulse on the board, sets this bit, and clears all other bits in this register. Writing 00000001x to this register clears this bit, as does a power up initiat¬ ed cold reset. This bit is NOT cleared by any warm reset

0 Cold Reset due to Power up

When reading this register, bit 0 =1 indicates that the board has received a power-up gen¬ erated cold reset since this bit was last cleared. This bit is cleared by writing OOOOOOOOx to this register or by a software initiated cold reset. When this bit is set by a power up reset all other bits in this register are cleared. This bit is NOT cleared by any warm reset.

12.3.4 Bus Interface State

Type: Read/Set/Clear Offset [22:0]: 7FFFE0

Present on all Xbus boards

Cold and warm reset affect this register as documented below Compatibility: subset/super-set

This is a Read Set/Clear register; commands are performed by writing to this register with data bits [4:0) pointing to a bit to set or clear and bit 7 determining whether the bit is set (=1 ) or cleared (=0). Reading this register returns 32-bits.

Note that since this register always returns 1s in bit positions [3:1], reading this register will always produce a non-zero resutt assuming the board is not broken. Stratabus boards used the ID PROM tor this purpose. It is recommended that this register be used for tiiat purpose on Xbus systems due to the large latency of accessing the IDPROM,.

31:23 These read bits/write encodings are reserved for additional board specific purposes. These bit positions always return O's when read unless assigned a board specific purpose.

22 Force Peer-to-Peer Mode - When this bit is set the operations in the second column of ta¬ ble 10, Peer to Peer and Non Peer-to-Peer Operations, will be turned into peer-to- peer cycle. Refer to section 6.8 for details. Warm or cold reset clears this bit.

21 Break PCIB board on backplane failure

When a failure on the backplane brings down a bus, it is necessary to break one of the two boards connected to it. The default is to break the CPU. When this bit is set, the PCIB board is the one designated for breaking. This Polo feature is described in full detail in section 6.4, Bus Errors, on page 44. This bit is cleared by cold reset

20 Enhanced EFQ/RWQ Nesting Disable - this mode is not implemented in Polo (Polo doesnt support enhanced nesting) • writes are ignored, reads return zeros.

19 EFQ Freeze State

This bit only exists on CPU boards. When set, the board busies any Xbus transaction de¬ coded for the Cyclops EFQ (Eviction Flush Queue) EXCEPT if it is a data return of any size. When read, this bit will not appear set until the Cyclops's EFQ is empty or is currently being emptied onto the Ibus. It is set in the following cases:

Set when an Xbus or local write ot 00000093x is performed AND the board's Freeze State bit is NOT set

29 December 1995 . , ,„ 90

Polo Software Programming Guide Stratus Company Confidential

Set when an Xbus or local write of 00000093x is performed AND the board's Freeze State bit is set AND all Cyclops and Cougar incoming R/W pipes are empty.

This bit is cleared by writing 00000013x, or by warm or cold reset. It is not cleared as a re¬ sult of the board going broken.

18 Break Even Board on TA Failure

This bit only exists on duplexable boards. Setting this bit causes the even board to break when a TA failure is detected; otherwise the odd board breaks. Software should set this bit so that the most recently added board of a pair is the one that breaks during a TA failure. The bit is set by writing 00000092x and cleared by writing 00000012x to this register. Warm reset has no effect on this bit, cold reset clears it. This bit always returns 0 on those boards which do not implement it

17 Simplexed mode - this mode is not implemented in Polo - writes are ignored, reads return zeros.

16 Nested OCU Op Disable - this mode is not implemented in Polo (Polo doesn't support nested OCU operations) - writes are ignored, reads return zeros.

15 On-line For Dumping

This bit only exists on some boards which have system memory. Setting this bit permits the memory from an otherwise off-line board to be read. The bit is set by writing OOOOOOδFx and cleared by writing OOOOOOOFx to this register. Cold and Warm reset clear this bit This bit always returns 0 on those boards which do not implement it.

1 Freeze State

This bit only exists on some boards which can be duplexed. On CPU boards, it is used in conjunction with the EFQ Freeze Mode bit As outlined in detail in table 15 on page 72 in the Xbus Functional Specification, writing to set this bit (0000008Ex) causes a board to busy all accesses directed to it except those from its partner until the board passes through the sync point. Writing OOOOOOOEx, Warm and cold reset cold reset clear this bit. This bit always returns 0 on those boards which do not implement it.

13 Update Mode

This bit only exists on boards which have system memory and lock cycles. As outlined in detail in table 15 on page 72 in the Xbus Functional Specification, when set this bit in¬ dicates that the board is the "update" mode, a mode used during the board synchro¬ nization procedure to update memory on a new board. When in this mode the on-line board sends its otherwise local writes to the Xbus and the new partner board (in the off-line update mode) updates its memory from these writes. Writing to set this bit (0000008Dx) puts a board in update mode. Writing OOOOOOODx, Warm and cold reset cold reset clear this bit. This bit always returns 0 on those boards which do not imple¬ ment it

12 Any CPU On-line

This bit indicates that some CPU board in the system is on-line. It reflects the state of the cpu online backplane signal and exists on all cpu/memory boards in the system. This bit always returns O's on IO boards. This bit is cleared whenever no CPUs are on-line (e.g. a single on-line CPU going off-line or a by a system-wide cold reset that brings all boards off-line). Writing to set (0000008Cx) or clear (OOOOOOOCx) this bit has no effect

11 First CPU On-line

This bit exists on CPU boards only and is set if this board becomes the first in a system af-

29 December 1995 91 3

Polo Software Programming Guide Stratus Company Confidential

ter a cold reset to gain mastership by winning an arbitration cycle and being the first to pass through the sync point. This bit is cleared by both cold reset warm reset. This bit always returns 0 on non-CPU boards.

10 Leave Sync Point Jump Switch

Writing to set ant one of these bits (00000086x, 00000087x, 00000088x, 00000089x, or OOOOOOδAx respectively) clears the other bits and instructs the board what state it should leave the sync point in. Regardless of the state of these bits, any CPU board which is the first on-line will leave the sync point simplex and clear all these bits. Ex¬ cept for the first on-line case, if none of these bits are set a board will not leave the sync point.

These bits can be cleared by writing 00000006x, 00000007x, OOOOOOOSx, 00000009x, or OOOOOOOAx respectively. Warm reset has no effect on these bits, cold reset clears them. Bits 7 and 8 exist only on boards which can be duplexed and always returns Os on all other boards, bit 9 exists only on boards with system memory that can be put on-line as just memory boards and always returns Os on other boards, bit 10 exists only on boards which can "jump switch" and always returns Os on all other boards.

A board reaching the sync point with bit 6 (Leave Sync Point Simplex) set will leave the sync point simplex. Bits 7 and 8 (Leave Sync Point Fast Duplex and Leave Sync Point Full Duplex) functionally are the same. A board will leave the sync point in a du¬ plexed state only if its partner is also at the sync point with one of these bits set. Two bits exist to provide software a differentiation between a last" duplex (one where two initialize but off-line boards are duplexed) and a lull" duplex (one where an off-line board is being synchronized to an on-line board). Bit 9 (Leave Sync Point On-line For Dumping) is for a board to leave the sync point with only its memory on-line, say for a memory dump. Note the Xbus architecture puts the system at risk of crashing when any simplex system memory is utilized. Bit 10 (Leave Sync Point Jump Switch) is to support the jump switch feature wherein the on-line board goes off-line and is re¬ placed by the previously off-line board.

Fore more information on these bits and there effect on board state see the Board Sync Register on page 93.

9 Leave Sync, point on line for dumping - not implemented in Polo systems.

8 Leave Sync Point Full Duplex

7 Leave Sync Point Fast Duplex

6 Leave Sync, point simplex - not implemented in Polo systems.

5 Duplexed

This bit only exists on duplexable boards. As outlined in detail in table 15 on page 72 in the Xbus Functional Specification, when set this bit indicates that the board and its partner are in lockstep. This bit is set as a result of the boards leaving the sync point in lockstep. The bit is cleared by cold and warm board resets, T/A failure, or by a board or its partner going Off-line or Broken. Writing this bit (00000085x or 00000005x) has no effect. This bit always returns Os on no-duplexabie boards.

4 On-line

This bit equals 1 if the board is on-line. As outlined in detail in table 15 on page 72 in the Xbus Functional Specification, an on-line board is fully functioning from the system perspective. Among other things, this means the board responds to all bus transac¬ tions directed toward it, including paired reads. Writing to set this bit (00000084x) or passing though the sync point (in some cases) brings a board on-line. Writing to clear this bit (00000004x) and the board breaking bring it off-line. Both warm reset and cold

29 December 1995 92

Polo Software Programming Guide oiratus Company Confidential

reset clear this bit.

3 Obey Both

The Xbus changes obeys based on the source of a transaction, not in response to errors. There is no need to set the obey state via software control. To maintain backwards compatibility with the Xbus, writes to the obey bits have no effect and reads always return 1.

2 Obey B - not implemented, returns 1 on reads.

1 Obey A - not implemented, returns 1 on reads.

0 Broken

This is a status bit used to indicate a hardware fault which has caused the two sides of the board to behave differently. This bit is set automatically by hardware or by the writing 00000080x to this register. This bit, as well as the diagnostic broken bit in the Test Control register, are cleared at the end of a cold or warm reset if the two sides are equal or by writing OOOOOOOOx to this register. Resets are the preferred method for unbreaking a board. When this bit is set the board cannot drive any external, (Xbus or C-connector), signals. The board will still respond to writes addressed to it in I/O space, (if capable), and will continue to check data and address information which would normally be transmitted to the bus. Whenever this bit is set by hardware a dou¬ ble bus error will be issued to the bus to abort any potentially bad information that may have been driven by this board. The act of clearing this bit generates a mainte¬ nance interrupt.

12.3.5 Board Sync Register

Type: Read Only Offset [22:0]: 7FFFD8

Present only on some duplexable Xbus boards Cold and warm reset do not directly affect this register Compatibility: exact

WARNING: Reading this register may return non-deterministic data and can be different side-to-side or board-to-board; it is important to discard whatever data is returned as soon as possible. Data left in registers can be pushed onto stack frames thus causing system crashes if different

A board reads this register to pass through the sync point This data return may be delayed by hardware and may have side effects. Reads of this register locally will cause the board to wait at the sync point Reads to this register from Xbus IO will return Os immediately and no sync procedure will take place.

It is possible to perform non-local IO accesses to from a board waiting at the sync point

The specific actions resulting from a board passing through the sync point (i.e. reading this register locally) depend on the state of the board and system involved:

1 First CPU On-line: The first CPU board in a system to try to read the sync register (or a the first CPU to win arbitration should multiple CPUs simultaneously attempt this) will pass through the sync point (i.e. have data returned to it when reading this register) and go on-line in a simplex state. Hardware then causes that board to assert the cpu_online backplane signal, thereby notifying other CPU boards that one CPU board has passed through the sync point.

Should the cpu_online backplane signal deactivate (say by the only on-line CPU 29 December 1995 _ 93

Polo Software Programming Guide Stratus company Confidential

breaking), another CPU board sitting at the sync point will be released in a similar fashion.

Note that a board passing through the sync point in this fashion will leave the sync point in a simplexed fashion regardless of the state of ti e "Leave Sync Point Sim¬ plex", "Leave Sync Point Fast Duplex", "Leave Sync Point Full Duplex", "Leave Sync Point On-line For Dumping", and "Leave Sync Point Jump Switch" bits in the Bus In¬ terface State register. Should any of those bits have been set, they are cleared in this case, thereby providing a mechanism for software to detect that the sync point was passed through as first cpu on-line.

2 Duplexed: Should a board's "Leave Sync Point Full Duplexed' or "Leave Sync Point Fast

Duplexed' bit in its Bus Interface State register be set, it will leave the sync point in lockstep with its partner when its partner reaches the sync point (assuming the part¬ ner's not already there) and a read is performed of the sync register and that board's "Leave Sync Point Full Duplexed" or "Leave Sync Point Fast Duplexed' bits are set. If a board is waiting at the sync point and its partner breaks, it will leave the sync point On-iine/Simplexed. (Actually, the duplexed bit will be set and then cleared immediate¬ ly when the hardware detects the partner is not on-line). Any other condition and the board will remain stalled at the sync point.

A pair of boards can pass through the sync point duplexed when both are new to the system (a "Fast Duplex") or when one board is already part of the system and a new board is being synchronized to it (a "Full duplex"). In the former case, the boards were both in the off-line state prior to passing through the sync point (but there was another CPU on-line, allowing the boards to stall at the sync point), in the latter case the "old ¹ board was in the On-line/Update/Freeze state and the "new" board was in the Off-line Update/Freeze state.

It is possible to put a board on-line duplexed when a board is already waiting at the sync point. This is done by setting the "Leave Sync Point Full Duplexed or Leave Sync Point Fast Duplexed' bit on while the board is stalled at the sync point.

3 On-line/Slmplexed: Should a board's "Leave Sync Point Simplexed' bit in its Bus Inter¬ face State register be set, it will leave the sync point, on-line but never duplexed. If it's partner is on-line, It will stall until its partner is off-line. Note, If a board and its part¬ ner both perform a go-on-line-simpiex at approximately the same time they will both go on-line but not duplexed, lockstep is not guaranteed after leaving the sync point. This condition is not supported on duplexable boards and Xbus transactions will con¬ flict with each other because they will both be using the same TRID.

If a board's partner is at the sync point with a the "Leave Sync Point Full Duplexed" or "Leave Sync Point Fast Duplexed' or "Leave Sync Point Jump Switch" bit set. the board will pass through the sync point simplexed and not effect its partner in any way. (The partner will remain at the sync point).

It is possible to put a board on-line simplexed when a board is already waiting at the sync point. This is done by setting the "Leave Sync Point Simplexetf' bit on while tiie board is stalled at the sync point

4 On-line For Dumping: Should a board's "Leave Sync Point On-line For Dumping" bit in its Bus interface State register be set it will leave the sync point immediately with its memory on-line but otherwise simplex and off-line. It is planned that this mode be used to support memory dumps. A board and its partner can be both "on-line for dumping" but Xbus transactions initiated by these boards are not supported

29 December 1995 , ^n 94

Polo Software Programming Guide Stratus Company Confidential

It is possible to put a board On-line For Dumping when a board is already waiting at the sync point. This is done by setting the "Leave Sync Point On-line For Dumping" bit on white the board is stalled at the sync point.

Jump Switch: Should a board's "Leave Sync Point Jump Switch" bit in its Bus Interface State register be set, it will either leave the sync point on-line and simplex or off-line, depending on whether it was on-line or off-line prior to passing through the sync point (If it was on-line it leaves off-line and if it was off-line, it leaves on-line.) It is planned that this mode be used to support on-line upgrade of board with different processor speeds or memory sizes.

12.3.6 LED Control

Type: Read/Write Offset [22:0]: 7FFFD0

Present on all Xbus boards

Cold and warm reset affect this register as documented below

Compatibility: exact

This register contains status/control of the LEDs on the board. There are three LEDs (red, yellow, green) arranged like a stoplight on each board handle. The following table indicates their states (disks and dumb devices are included for completeness). For more information on the LEDs, their purpose and software model, see the appropriate sections of the maintenance and diagnostic specification.

Table 16. LED States

31:3 Reserved

These bits are reserved for future use. Writes to the register will have no effect. Reads of this register will return zeros in these bits.

Red LED State

The red LED is turned on by hardware whenever the board is in the broken state. When

29 December 1995 95

^

Polo Software Programming Guide Stratus Company Confidential

the board is not broken, writing this bit can turn on (wnte a 1 ) or off (write a 0) the LED. Note that hardware will not clear the LED when the board transitions from bro¬ ken to not broken; that must be done by software. The red LED is set by warm and cold reset.

2 Yellow LED State

The yellow LED on is used to indicate that a board may not be removed. It is transition- driven: on duplexable boards, it is turned on by hardware when the board transitions from off-line to on-line-simplexed, or from on-line-duplexed to on-iine-simplexed (due to its partner breaking). It is turned off by hardware on the transition to duplexed or broken; it is not cleared by the on-lιne->off-iιne transition. On non-duplexable boards, the LED is set when the go on-line command is issued. The yellow LED may be set or cleared by software at any time. The yellow LED is cleared by warm and cold re¬ set.

0 Green LED State

The green LED is turned on by hardware when the go on-line command is issued, and is turned off when the board goes off-line or broken. These actions are state transition driven. Only the transition causes hardware to set or clear the light. At any time, the software may also set or clear the green led. The green LED is cleared by warm and cold reset.

12.3.7 Slot ID

Type: Read Only Offset [22:0]: 7FFFC8

Present on all Xbus boards

Cold and Warm Reset have no effect on this register

Compatibility: exact

This register contains the 4 bits that indicate the physical slot position of this board. This is the slot number address that this board will respond to for non-global I/O accesses. Note that this register is readable locally, when not duplexed (e.g. during diagnostics).

The 4 slot id bits are located in bits 3:0 of the slot id register. Bits 31.4 are unused and will return zeros when read.

12.3.8 Read Ping Interval

Type: Read/Write Offset [22:0]: 7FFFC0

Present on all Xbus based boards that can issue I/O reads Cold reset clears this register, warm reset has no effect Compatibility: exact

31:1 Reserved

0 Read Ping Interval

The value loaded into this bit determines the amount of time a read will be outstanding be¬ fore a "Ping" bus transaction is sent on the Xbus to determine if there is still a target that intends to respond to the request. Wnttng a one sets the read ping interval to 31 bus phases, or 2.667 microseconds. This value is provided for diagnostic and simula¬ tion use only. Writing a zero sets the interval to 99.416 microseconds, or 1, 193 bus phases (the default). Cold reset clears this register; in normal operation there is no need to touch this register.

This register is not implemented on PCIB boards (since they cannot issue I/O reads) When a read

29 December 1995 96

Polo Software Programming Guide tratus Company Confidential

request is committed on the Xbus, i.e. the request is not terminated with BUSY or ERROR, a time¬ out counter which was pre-loaded with either 31 or 1 ,193 is enabled. There is a single counter that times the duration of the oldest request from a board that is outstanding; if a response is not received before this counter reaches terminal count, a ping transaction is sent on the bus. Should no acknowledge to the ping be received, the bus interface returns zeros and an error indicator to the requestor. This error indication is a low priority machine check on the CPU board. If an acknowledge to the ping operation is received, the time-out counter is reloaded according to the contents of this register and the timeout count is restarted. If a response is returned before the counter reaches terminal count, date is returned in the usual fashion.

A read of this register provides the most recently written data. Bits 31 :1 are unused and will return zeros when read.

12.3.9 General Purpose Communications [7:0]

Type: Read Write Offset [22:0]: 7FFF98:7FFF60

At least 4 present on all Xbus based boards, up to 8 on some. Cold and warm reset affect these registers as documented below Compatibility: exact

These registers are provided for any general communication desired by any resource on or off the board. Because boards may have different needs that are not defined at this time, eight General Purpose Communication registers are provided in IO space. Not all 8 registers are necessarily implemented on all bus ASICs, but each board is required to have at least the first four (3:0).

Table 17. General Purpose Registers

12.3.10 Memory Size/Location

Type: Read Write Offset [22:0]: 7FFF58

Present only on Boards with Global System Memory Cold reset clears this register, warm reset has no effect Compatibility: implemented for software compatibility only

31:8 reserved

Writes have no effect; reads return zeros.

7:0 These bits are implemented for software compatibility with Jetta only. The bits are read/write but have no effect on operation.

12.3.11 Test Control

Type: Read/Set/Clear Offset [22:0]: 7FFF48

29 December 1995 97

^\ ^ζ

Polo Software Programming Guide Stratus company Confidential

Present on all Xbus boards

Cold reset clears this register, warm reset has no effect

Compatibility: subset

This is a Read/Set/Clear register, commands are performed by writing to this register with date bits [4:0] pointing to a bit to set or clear and bit 7 determining whether the bit is set (=1 ) or cleared (-0). Reading this register returns 32-bits.

31:7 Reserved.

6 D-side Difference Bit

When the D-side Difference Enable is set, reading this bit returns a 1 from the D side hard¬ ware, and a 0 from the C side. This C-D difference can be used for testing compara¬ tors internal to the board. This bit is cleared by cold reset.

5 D-side Difference Enable

Setting this bit enables the D-side difference bit. When it is enabled, the D-side difference bit becomes a one on the D side bus ASIC, and remains a 0 in the C-side ASIC. This bit is cleared by cold reset.

WARNING: Setting this bit and then reading this register will return different data on C vs. D sides of the board (via bit 6 below). This will break you if not read local¬ ly.

4 Diagnostic.Broken

This bit provided to facilitate self-test This bit is set whenever a condition that would set the Broken bit of the Bus Interface State register occurs. However, it may be cleared inde¬ pendently of the Broken Bit by writing 0000004x to this register. This permits testing of various board features while the board remains Broken, preventing the testing from pol¬ luting the Golfbus. This bit is also cleared by cold reset. Warm reset sets this bit, be¬ cause it causes the board to go broken for the duration of the reset Writing 00000084x (which normally would set a bit) have no effect on this register. Clearing Broken (by writ¬ ing to the Bus Interface State register) does not clear Diagnostic Broken.

Setting this bit enables the D-side difference bit. When it is enabled, the D-side difference bit becomes a one on the D side bus ASIC, and remains a 0 in the C-side ASIC. This bit is cleared by cold reset.

3:0 Reserved.

12.3.12 Bus Interface Fault Reporting

Type: Read/Set/Clear Offset [22:0]: 7FFF38

Present on all Xbus boards

Bit 1 set by cold and warm reset; for other bits cold reset clears this bit, warm reset has no effect

CD Different: read via unpaired space

Compatibility: incompatible

This register controls the error latching and reporting around bus faults, it also is used to rearm the broken registers. There are five error registers: Bus Info Error Status, Misc. Error Status, Bus Error Byte Status, Control Bus Error Status, and Voter Error Transceiver Status. All are latched when an error is detected in the system. This register provides a means of disabling certain faults from causing the "Error Latching" to occur, as a way of allowing new faults to be detected in the presence of known, continuous faults. Maintenance interrupts are generated on the latching of a fault; if a fault is disabled from causing error latching, it is also disabled from causing a

29 December 1995 98 ^{" "}

Polo Software Programming Guide stratus Company Confidential

maintenance interrupt An overview of the information supplied and how it is intended to be used is provided in section 9.4, Error Reporting, on page 98.

Note that this register is both Read Set/Clear and CD different. A single write to set or clear a bit affects the C and D side bits, thus to set bits 3,11 ,19, and 27, it is only necessary to write 00000083x. It is not possible to write only the C or D side.

These bits indicate whether the various classes of faults cause the Error Registers to record stete ("Error Latching") and generate a maintenance interrupt. The default is all zeros, i.e. any fault causes a maintenance interrupt and all of the Error Registers to Update. Error Latched on Thermal Faults Disabled suppresses Error Latching and the resulting maintenance interrupt due to Thermal faults. However, if another class of fault occurs and a thermal fault is present, the thermal fault will be present in the updated error register. Likewise, Error Latched on Incoming 3-Way Vote Fault Disabled, Error Latched on Control Bus Fault Disabled, and Error Latched on Clock Faults Disabled determine whether 3-way voter, Control bus errors, or clock faults trigger Error Latching and maintenance interrupts.

'Error Latched on TA Problem Disabled (bits 5,21 ,13,29) is named "Error Latched on non-OBEYed Bus Fault Disabled" on Golfbus systems. In Golfbus systems, this bit has two functions: disabling error latching due to errors on the non-obeyed bus, and due to TA problem. In the Xbus system, only the TA problem functionality is retained, so the name of the bit has been changed for clarity.

31,15 D side Error Latched on Control Bus Faults Disabled 30,14 D side Error Latched on Clock Faults Disabled - CPU only

Clock faults only occur on CPU boards. 29,13 D side Error Latched on TA Problem Disabled ^* - CPU only

TA faults only occur on CPU boards.

28,12 D side Error Latched on Incoming 3-Way Vote Fault Disabled

27,11 D side Error Latched on Thermal Faults Disabled - CPU only Thermal faults only occur on CPU boards.

26,10 D side Board Logic Maint Int

This bit being set indicates that a maintenance interrupt was issued by the logic (excluding the other bus interface) on this board. The appropriate on board register(s) should be examined to determine the source of the interrupt This bit is cleared by cold reset or by writing to clear this bit (00000002x). This bit may be set (and a maintenance inter¬ rupt generated) by writing 00000082x. Note: For Gambit ASIC, whenever the Gam¬ bit specific logic sets this bit, bits 18 and 2(C side Board Logic Maint int) also get set

25,9 D side Bus Interface Maint Int

This bit being set indicates that a maintenance interrupt was issued from this bus interface (including the clock chip and board logic maint int). Writing to set this bit causes a maintenance interrupt to be pulsed on the bus though the read bit in this register re¬ mains set until cleared. This bit is cleared by writing 00000001x to this register. Since both warm and cold reset break and then unbreak a board (events which generate maintenance interrupts), either warm or cold reset leave this bit set.

24,8 D side Error Data Recorded Rearm Error and Broken Registers

When this bit =1, the following registers contain new information: Bus Info Error Status, Misc. Error Status, Bus Error Byte Status, Control Bus Error Status, and Voter Error Transceiver Status. Writing a 0 to this bit clears these registers and "rearms" them so

29 December 1995 , - 99

A S }

Polo Software Programming Guide Stratus Company Confidential

that they will capture the state of the next applicable error condition. Clearing this bit also "rearms" the Common Broken Status and ASIC Specific Broken Status registers Wπting a 1 to this bit has no effect

23.7 C side Error Latched on Control Bus Faults Disabled

22.6 C side Error Latched on Clock Faults Disabled - CPU only

21.5 C side Error Latched on TA Problem Disabled ^* - CPU only

20.4 C side Error Latched on Incoming 3-Way Vote Fault Disabled

19.3 C side Error Latched on Thermal Faults Disabled - CPU only

18.2 C side Board Logic Maint Int Note: For Gambit ASIC, whenever the Gambit specific logic sets this bit, bits 26 and 10(D side Board Logic Maint int) also get set.

17.1 C side Bus Interface Maint Int

16.0 C side Error Data Recorded/Rearm Error and Broken Registers

12.3.13 Common Broken Status

Type: Read Only Offset [22:0] : 7FFF30

Present on all Xbus boards

Cold reset clears this register, warm reset has no effect

CD Different: read via unpaired space

Compatibility: incompatible

This register contains status bits from the broken logic comparators that are common to all Xbus boards. It is cleared by cold reset (unless the board leaves cold reset broken). The contents of this register are frozen when a board first goes broken, so that the only comparators that caused the broken condition are flagged. This is so that comparators that miscompare further down the line (after the error has rippled through the logic) do not obscure the cause of the original failure. This register is re-armed via the Re-arm Error registers bit in the Bus Interface Fault Reporting register.

31,15 D-side saw control bus set broken

30, 14 D-side saw D LL out-of-lock

29,13 D-side saw dnve miscompare error on a or b bus (i.e. A C to D side info out miscompare)

28,12 D-side saw C-side bus ASIC set broken

27,11 D-side board logic (external to the bus ASIC) generated broken

26,10 D-side slot parity error

25,9 D-side ASIC saw drive miscompare error on 3-way voted line.

24.8 D-side Software set broken by command

23.7 C-side saw control bus set broken

22.6 C-side saw DLL out-of-lock

21.5 C-side saw drive miscompare error on A or B bus (i.e. a C to D side info out miscompare)

20.4 C-side saw D-side bus ASIC set broken

19.3 C-side board logic (external to the bus ASIC) generated broken

18.2 C-stde slot panty error

17.1 C-side ASIC saw drive miscompare error on 3-way voted line.

29 December 1995 100

Polo Software Programming Guide Stratus Company Confidential

16.0 C-side Software set broken by command

12.3.14 ASIC Specific Broken Status

Type: Read Only Offset [22:0]: 7FFF28

Present on all Xbus boards

Cold reset clears this register, warm reset has no effect

CD Different: read via unpaired space

Compatibility: incompatible

This register conteins status bits from the broken logic comparators that are specific to a particular Xbus ASIC. It is cleared by cold reset (unless the board leaves cold reset broken). The contents of this register are frozen when a board first goes broken, so that the only comparators that caused the broken condition are flagged. This is so that comparators that miscompare further down the line (after the error has rippled through the logic) do not obscure the cause of the original failure. This register is re-armed via the Re-arm Error registers bit in the Bus Interface Fault Reporting register.

CPU ASIC (Cyclops): 31,15 Reserved

30.14 D-side Fan broken

29.13 D-side Sable broken

28.12 D-side Cougar broken

27.11 D-side TA broken

26.10 D-side IM OK broken

25.9 D-side Arbitrary broken.

24.8 D-side Heuristic broken 23,7 Reserved

22,6 D-side Fan broken

21,5 C-side Sable broken

20,4 C-side Cougar broken

19,3 C-side TA broken

18,2 C-side IM OK broken

17.1 C-side Arbitrary broken. 16,0 C-side Heuristic broken PCIB ASIC (Gambit):

31.15 Reserved

30.14 Reserved

29.13 Reserved

28.12 Reserved

27.11 D-side Gambit specific

26.10 D-side Xbus Parity generator fault

25.9 D-side Arbitrary broken.

29 December 1995 - 101

Polo Software Programming Guide Stratus Company Confidential

24.8 D-side Heuristic broken 23:7 Reserved.

22:6 Reserved.

21:5 Reserved.

20:4 Reserved.

19.3 C-side Gambit specific

18.2 C-side Xbus Parity generator fault 17,1 C-side Arbitrary broken.

16.0 C-side Heuristic broken

12.3.15 Bus info Error Status

Type: Read Only Offset [22:0]: 7FFF20

Present on all Xbus boards

Cold reset clears this register, warm reset has no effect

CD Different: read via unpaired space

Compatibility: subset

Any bus error signaled by the system, or a voter, control bus, thermal, or clock error on this board causes this register (and the other Error Latching registers) to freeze with state recorded for the offending info phase. This includes both non-fatal and fatal (i.e. board breaking) errors. All bits in this register correspond to the same info phase. Cold reset clears this register.

31,15 D-side some other board drove a the Bus B Error backplane signal and I did not.

30,14 reserved

29,13 D-side The bus error whose info was recorded was signaled by a different board

28,12 reserved

27,11 D-side ASIC saw parity error on B bus

26,10 D-side ASIC saw parity error on A bus

25.9 D-side ASIC saw loopback fault on the B bus on the info bits the D-side drove. 24,8 D-side ASIC saw loopback fault on the A bus on the info bits the D-side drove. 23,7 C-side some other board drove a the Bus A Error backplane signal and I did not 22,6 reserved

21,5 C-side This board was bus master for the info phase recorded

20.4 reserved

19.3 C-side ASIC saw parity error on B bus 18,2 C-side ASIC saw parity error on A bus

17.1 C-side ASIC saw loopback fault on the B bus on the info bits the C-side drove. 16,0 C-side ASIC saw loopback fault on the A bus on the info bits the C-side drove.

This register reports the various bus and compare faults separately for the C side Bus ASIC and the D-side ASIC.

29 December 1995 , 102

Polo Software Programming Guide iratus Company Confidential

This register will not accurately reflect the state of the errors if the board is in the "funny" state. Since boards in the "funny" state do not enter the error sequence, it is not possible to track the errors.

All bits are updated upon the detection of an error and remain at the same stete until the Error Date Recorded/Rearm Error Registers bit of the Bus Interface Fault Reporting Register is cleared.

12.3.16 Misc. Error Status

Type: Read Only Offset [22:0]: 7FFF18

Present on all Xbus boards

Cold reset clears all bits except the Thermal faults bits.

Thermal fault is unaffected by any resets.

Warm reset has no effect on this register.

CD Different: read via unpaired space

Compatibility: exact

The bit definitions for this register are identical to the Golfbus Specification definitions with one difference: thermal and clock faults in Polo only occur on CPU boards so the bits associated with those faults are CPU only.

28,12 D-side T/A problem (CPU/MEM board ONLY.)

27,11 D-side thermal fault bit 0 - under temperature (CPU/MEM board ONLY.)

26,10 D-side clock recovery chip status bit 1 (CPU/MEM board ONLY.)

25,9 D-side clock recovery chip stetus bit 0 (CPU/MEM board ONLY.)

24,8 D-side ASIC saw a three way voter failure

23,7 Reserved.

22,6 Reserved.

21,5 Reserved.

20,4 C-side T/A problem (CPU/MEM board ONLY.)

19,3 C-side thermal fault bit 1 - over temperature (CPU/MEM board ONLY.)

18,2 C-side clock recovery chip status bit 1 (CPU/MEM board ONLY.)

17,1 C-side clock recovery chip status bit 0 (CPU/MEM board ONLY.)

16,0 C-side ASIC saw a three way voter failure

There are two clock recovery chips on every board; each recovery chip has two pins which indicate the status of the Backplane fault tolerant clock. The various combinations of these status bits are indicated in table 18

The clock status bits will change off the falling edge of 24 MHz, therefore it is necessary for the Bus Interface ASIC's to latch the stetus bits on the rising edge of 24 Mhz ( phase 12_3) to insure

29 December 1995 103 l

Polo Software Programming Guide Stratus company Confidential

proper setup and hold time.

Table 18. Clock Status Definition

Any change on a Thermal fault bit results in a maintenance interrupt (unless disabled) and freezes all registers. These two bits are from the thermal sensor on the board. bit'O] Represents the under temperature pιn(6) on the thermal sensor, (etch run to D ASIC.) bit[1] Represents the over temperature pιn(7) on the thermal sensor, (etch run to C ASIC.)

Table 19. Thermal fault Status Definition

The T/A problem bit (Cyclops only bit) when set, represents that there is a non-fatal problem in the T/A Hardware. The board or backplane could be at fault. This condition should be cleared up. If left un-fixed, REAL T/A problems will not be handled correctly. This bit is masked off by the "Error Latched on non-OBEYed Bus Fault Disabled' bit

12.3.17 Control Bus Error Status Type: Read Only Offset [22:0]: 7FFF10 Present on all Xbus boards

Cold reset clears this register, warm reset has no effect CD Different: read via unpaired space Compatibility: New Register

This register records the control bus error status to aid in fault isolation of a system experiencing single or multiple bit errors on the control buses. All bits are updated upon any bus error signaled by the system, or a voter, control bus, thermal, or clock error on this board, and remain at the same state until the Error Data Recorded/Rearm Error Registers bit of the Bus Interface Fault Reporting register is cleared. Cold reset clears these registers.

The bits are set for both single bit (correctable) and double bit (detectable, causes board broken) errors. Check the Common Broken Status register to see if the error caused board broken.

31,15 Reserved

29 December 1995 104

Polo Software Programming Guide Stratus Company Confidential

30,14 Reserved

29.13 D-skte saw error on coπtrol_out_p

28.12 D-side saw error on coπtrol_in_p

27.11 D-side saw error on control_out_o

26.10 D-side saw error on control_in_o

25.9 D-side saw error on control_put_n

24.8 D-side saw error on control_in_n 23,7 reserved

22.7 C-side arbitration out of sync with D-side.

This bit is set when the C side checking logic sees an unexpected value on the bus_req control line driven by the D-side.

21 ,5 C-side saw error on control_out_p

20,4 C-side saw error on control_in_p

19,3 C-side saw error on control_out_o

18,2 C-side saw error on controlj n_o

17,1 C-side saw error on control_put_n

16,0 C-side saw error on control_in_n

12.3.18 Bus Error Byte Status

Type: Read Only Offset [22:0]: 7FFF08

Present on all Xbus boards

Cold reset clears this register, warm reset has no effect

CD Different: read via unpaired space

Compatibility: usage change

This register records the bus error status on a per-byte basis to facilitate debug. All bits are updated upon any bus error signaled by the system, or a voter, control bus, thermal, or clock error on this board, and remain at the same stete until the Error Data Recorded Rearm Error Registers bit of the Bus Interface Fault Reporting register is cleared. Cold reset clears these registers.

The bits are inteφreted as follows: If the board was the bus master (indicated in the Bus Info Error Status register) at the time of the error, this register indicates which bytes had a loopback error. If the board was not a bus master at the time of the bus error, these bits will be zero.

31.14 Reserved 30,14 Reserved

29.13 D-side saw error on trid or fuπc_op

28.12 D-side saw error on parity

27.11 D-side saw error on info[31 A)

26.10 D-side saw error on iπfo(23: 16)

26.9 D-side saw error on info[15:8]

24.8 D-side saw error on info[7:0]

29 December 1995 , r -> 105 sΛ }

Polo Software Programming Guide Stratus company Confidential

23.7 Reserved

22.6 Reserved

21.5 C-side saw error on trid or func_op

20.4 C-side saw error on parity

19.3 C-side saw error on info[31 :24]

18.2 C-side saw error on info[23: 16]

17.1 C-side saw error on info[15:8]

16.0 C-side saw error on info[7:0]

12.3.19 Voter Error Transceiver Status

Type: Read Only Offset [22:0]: 7FFF00

Present on all Xbus boards

Cold reset clears this register, warm reset has no effect

CD Different: read via unpaired space

Compatibility: incompatible

These register records the status of the 3-way voting logic. All bits are updated upon any bus error signaled by the system, or a voter, thermal, or dock error on this board, and remain at the same state until the register is rearmed by the Error Data Recorded/Rearm Error Registers bit of the Bus Interface Fault Reporting register. Error latching and maintenance interrupts due to voter errors can be suppressed by the Error Latched on Incoming 3-Way Vote Fault Disabled bit in the Bus Interface Fault Reporting register.

For a list of which signals go through each transceiver, please see Xbus Voted Signal Partitioning on page 96.

CPU ASIC (Cyclops):

31 ,15 Reserved, read as zero.

30,14 Reserved, read as zero.

29,13 D-side saw error on onlinejn

28,12 D-side saw error on reset

11 , 27 D-side saw error on syncjn

26,10 D-side saw enor on board_not_broken_p

25,9 D-side saw error on board_not_broken_o

24.8 D-side saw error on board jιot_brokenjι

23.7 Reserved, read as zero.

22.6 Reserved, read as zero.

21.5 C-side saw error on onlinejn

20.4 C-side saw error on reset

19.3 C-side saw error on syncjn

18.2 C-side saw enor on boar _not_broken j _** *

17.1 C-side saw error on board jiotjorokenjD

29 December 1995 106

Polo Software Programming Guide stratus Company Confidential

16.0 C-side saw error on board_not_broken_n PCIB ASIC (Gambit): 31 ,15 Reserved, read as zero. 28,12 D-side saw error on reset_o 27,11 D-side saw error on resetji 26,10 D-side saw error on board_not_broken_p 25,9 D-side saw error on board_not_broken_o 24,8 D-side saw error on board_not_broken _n 23,7 Reserved, read as zero. 22,6 Reserved, read as zero. 21,5 Reserved, read as zero. 20,4 C-side saw error on reset_o 19,3 C-side saw error on resetji 18,2 C-side saw error on board jιot_brokenj>

17.1 C-side saw enor on board_not_broken_o 16,0 C-side saw error on board_not_broken_n

12.3.20 Bus ASIC Chip Revision Type: Read Only Offset [22:0]: 7FFEF8 Present on all Xbus boards Cold and Warm reset have no effect Compatibility: new values

This register returns a revision number of the ASIC. Note: this information should also be available from the ID PROM; it is included here to simplify tracking parts in the lab when the ID PROM is not stable.

Table 20. Bus ASIC Revision Numbers

12.3.21 Performance Counter Type: Read Write Offset [22:0]: 7FFEF0 Present on CPU/MEM boards Cold and warm reset Clear this register. Compatibility: exact

A read of this register will return the value of this 32 bit performance counter. The counter may be set to any value by writing to this register. This counter will increment every time the trigger condition (as defined in pert _counter_trig, perf_couπter_mask. perf_∞unter_trig_πot_equal and perf_counter_trig_enable) is encountered. This counter will never increment past the value of OFFFFFFFFx. If the counter were set to zero and the trigger condition was set to always

29 December 1995 107

Λ l> $

9781

Polo Software Programming Guide Stratus company Confidential

increment, the counter would reach the value of OFFFFFFFFx in 358.1 seconds (5.97 minutes). This register may be read and written while the counter is enabled

12.3.22 Performance Counter Trig[1 :0]

Type: Read/Write Offset [22:0]: 7FFEE8:7FFEE0

Present on CPU/MEM boards

Cold and warm reset clear these registers.

Compatibility: exact

This register defines the trigger condition that will increment the performance counter. A trigger condition is evaluated every 83.381361 ns (one -12MHZ Xbus phase). All Evaluations occur using data that is "lined up" with the given Xbus bus operation. That is, function codes, physical address, busy, error, ACK (etc.) are all delayed the correct amount so that all the information presented to the tngger circuitry have meaning for that Xbus operation. Any bit may be masked to a don't care using the perf_counter_mask registers (below).

See Perf_Counter_Mask for bit definitions.

12.3.23 Performance Counter Mask[1:0]

Type: Read/Write Offset [22:0]: 7FFED8:7FFED0

Present on CPU/MEM boards

Cold and warm reset clear these registers.

Compatibility: exact

This register works in conjunction with the perf_counter_tng register above. A bit that is a one '1 ' in this register signifies you "don't care" to evaluate this bit as part of the trigger. A bit that is a zero '0' in this register signifies you wish to evaluate this bit as part of the trigger.

Perf_Counter_Trig[0]: (7FFEE0) OR Perf_Counter_Mask[0]: (7FFED0)

31:26 Reserved.

26 xbusjerror: Some board drove A or B bus error tor this Xbus operation or this operation was aborted due to a previous bus error. All above data could be wrong/invalid

25 my_busy: This board drove busy for this Xbus operation.

24 xbus_busy: Some board drove busy for this Xbus operation.

23 myjack: This board drove ACK for this Xbus operation.

22 xbus_ack: Some board drove ACK for this Xbus operation.

21 ι_drove_bus: This board was bus master for this Xbus operation

20 fιrst_op: This is the first operation of a block transfer.

19 func op: The into bus contains a valid function code, byte enables, RC and physical ad ^¬ dress, else it ts date.

18:12 tnd[6:0]: The 7 bit trid used on this bus operation.

11:6 functιon_code (5:0]: Needs qualification with funcjop (bit 19).

5:4 remote coherent bits [1 :0]: Needs qualification with tunc_op (bit 19).

3:0 byte enables [3:0]: Needs qualification with funcjop (bit 19).

Perf_Counter_Trig[1]: (7FFEE8) OR Perf_Counter_Mask[1]: (7FFED8)

29 December 1995 108

Polo Software Programming Guide Stratus Company Confidential

31 :0 physical _address[31 :00]: Needs qualification with funcjop.

12.3.24 Performance Counter Control

Type: Read/Set/Clear Offset [22:0]: 7FFEC8

Present on CPU/MEM boards

Cold and warm reset clears this register

Compatibility: exact

This is a Read/Set/Clear register, commands are performed by writing to this register with data bits [4:0] pointing to a bit to set or clear and bit 7 determining whether the bit is set (=1 ) or cleared (=0). Reading this register returns 32-bits.

31:2 Reserved.

I Perf_Counter_Trig_(on)_Not_Equal

This bit returns a 1 if a board's performance counters are incrementing when the trigger condition is not equal, otherwise it returns a 0. Writing to set this bit (00000081 x) will cause the counter trigger circuitry to increment the pert_counter whenever the trigger condition (as defined in pert jcounter rig, perf_counter_mask) is NOT seen. Writing to clear this bit (0000001 x) will cause the perf_counter to increment when the trigger condition is seen. This bit is cleared on cold and warm reset

0 Perf_CounterJΞnable

This bit returns a 1 if a board's performance counter is enabled, otherwise it returns a 0. Writing to set this bit (00000080x) will enable the performance counters and the per¬ formance counters will increment every time the trigger condition (as defined in perf_counter_trig, perf_couπterjnask and perfjx>unterjrig_not_equal) is encoun¬ tered. Writing to clear this bit (OOOOOOOx) disables any incrementing of the counter. This bit is cleared on cold and warm reset.

12.3.25 Fault Bit[1:0]

Type: ReadWrite Offset [22:0]: 7FFEC0

Present on all Xbus boards 7FFEB8

Cold reset clears this register, warm reset has no effect Compatibility: new register

This register is used to generate errors on the Xbus or local to the board depending on the setting of the Error Control register. Each bit in these registers corresponds to a bit on the info bus and the control bus. Setting that bit in the register will cause the corresponding bit on the bus to be incorrect (inverted). The timing of when the bit is incorrect is based on the Error Control register setting.

Fault Bit[0]: (7FFEC0)

31:0 Fault corresponding bit on info bus 31 :0 for the first half of the phase

Fault Bit[1]: (7FFEB8)

24 Fault parity for the first half of the phase

23 Fault funcjop for the first half of the phase

22:16 Fault trid[7:0] for the first half of the phase

15:12 Fault corresponding to checkbits 7 :4 for second half of the phase

II Fault unused control[3]

29 December 1995 109

Polo Software Programming Guide Stratus Company Confidential

10 Fault busy

9 Fault maintjnt

8 Fault ACK

7:4 Fault corresponding to checkbits 7:4 for first half of the phase

3 Fault bus_err_b

2 Fault bus_err_a

1 Fault grant jnh

0 Fault bus.req

12.3.26 Data Match

Type: Read/Write Offset [22:0]: 7FFEB0

Present on all Xbus boards

Cold reset clears this register, warm reset has no effect Compatibility: new register

This register is used for the data matching capability of the error generation. This allows the user to set a trigger and the error will not occur until a match is seen on the incoming info bus.

31:0 Data pattem to match against incoming info 31 :0 for the first half of the phase

12.3.27 Error Control

Type: Read Set Offset [22:0]: 7FFEA8

Present on alt Xbus boards

Cold reset clears this register, warm reset has no effect Compatibility: new register

Note: The Error Control register is a set/clear register, software can set one bit at a time, but cannot dear any bits. Bits are cleared by hardware once the error has been generated; also once bit 0 is set no other bits can be changed by software because the error has already been initiated

This register is used to control the different types of errors seen on the system. There are three modes of operation. The first is normal mode, no error will be produced on the system. The second mode is immediate trigger. This causes an error after this register is written to and an access is directed to the Xbus. The last mode is data match. This allows a trigger pattem to be set in the Data Match register. The error will not occur until the pattem is seen on the incoming info bus. The matching is done on the first half of the info phase. Note: It is important to set up the Fault Bit[l :0] register and the Data Match register before this register is set. The types of errors that can be produced corresponds directly to the cases described in section 6.4, Bus Errors, on page 44.

To produce an error, the first step is to decide which type of error is desired from table 8, Error Types. Then, if the error is a Data Match error the Data Match register must be initialized along with the Fault Bit[1 :0] register. If the error is immediate then the Fault Bit[1 :0] register must be initialized. If a transient error or a bus busy is desired the Data Match register must be initialized but the Fault Bit(1 :0] register does not have to be initialized. After necessary registers are initialized the bit in the Error Control register should be set which corresponds to the error the user is looking for. Then bit 0 of the Error Control register is set. If it is an immediate error then as soon as the Xbus is accessed the error will occur. If the error is a date match error then it will not occur until the data pattem is seen on the incoming info bus.

29 December 1995 , 110

Polo Software Programming Guide airatus Company Confidential

Once bit 0 is set and the error has occurred hardware will clear the bit that indicates the error and it will also clear bit 0. The error will be asserted for as long as necessary to produce the particular case. Therefore some cases require the error to occur during both CPUTest and IOTest where as other cases will only require the fault to happen for one phase. Only one error is allowed with the exception of the transient fault. There can be a transient fault on both buses. If more than one bit is set, the lowest bit number will have priority. All bits will be cleared after the error is produced.

Table 21. Error Types

The following lines describe behavior that might not be expected with some of the error insertion cases.

Case 17: the transient bus fault will not cause the faulting board to freeze its error registers, all other boards will freeze their error registers

Case 18: busy will be asserted for two phases.

29 December 1995 111 s l ^

Polo Software Programming Guide Stratus company Confidential

12.4 CPU Specific Xbus Register Descriptions

This section deals with the ASIC specific Xbus/Gotfbus registers found in the P0I0/HSC2 ASICs. All of the registers in the board specific space with the exception of the I/O Address Map Error register are non-privileged registers. A description of all CPU specific Xbus registers can be found following the table.

Table 22. Cyclops/Mirage Specific Register Map

a. privileged space & - F; nαα-pnvileged space, & - E

Cyclops and Cougar (the other large ASIC on the CPU board) share the privileged page 7FF and the non-privileged page 7FE with Cougar registers in the lower half of the pages in 000->7F8 and Cyclops registers in the top half in 800 -> FF8. Table 23 illustrates how the addresses for either CPU Xbus specific or CPU Xbus common registers are formed.

Table 23. CPU Register System Address Formation

a. paired space -> - 1; unpaired space -> - 0

12.4.1 ASIC Specific Configuration

Type: Read Set/Clear Only in privileged page 7FF Offset [22:0]: 7FF830 Cold reset clears this register, warm reset has no effect

31:2 reserved

2 IOVA address mode 1 = 40 bit mode, 0 = 32 bit mode, refer to section 5.2.

1 Fan Fault set broken disabled

0 Maintenance Interrupt disabled for IOVA errors.

12.4.2 I/O Address Map Error 1

Type: Read Only in privileged page 7FF Offset [22:0]: 7FF828

Cold reset clears this register, warm reset has no effect

31:0 IOVA Address - refer to figure 21.

29 December 1995 112

Polo Software . •ogramming Guide όtratus Company Confidential

12.4.3 I/O Address Map Error 0

Type: Read Only in privileged page 7FF Offset [22:0]: 7FF820

Cold reset clears this register, warm reset has no effect

31:10 reserved.

10 IOVA Out of bounds Access Error

9 IIOVA Illegal Slot Error - PCI slot accessing (from Xbus TRID) disagrees with PCI slot field in the IOVA map entiy.

8 IOVA Checksum Error.

7 IOVA Invalid Map Entry Error.

6:0 IOVA Trid (Xbus trid during 1st bus cycle of IOVA info phase).

12.4.4 Time Of Day

Type: Read Write in Privileged page 7FF Offset [22:0]: 7FF800

Read only in non-Privileged page 7FE Offset [22:0]: 7FE800

Cold reset clears this register, warm reset has no effect

This register (and the following three) implement the user accessible time of day function on Mercury. These registers are initialized through Global writes to page 7FF, and are read-only in the user visible non-privileged page 7FE.

The time of day is maintained in a single 56 bit master counter that increments at 12MHz. For compatibility with existing Stratus software, various slices of this counter are referred to by different names. The time of day refers to the most significant 32 bits of the master counter: it gives the current time in seconds.

Figure 34. Cyclops Master Counter

This register should only be written when the counter is stopped (Master Counter Enable bit in the Jiffy Control register is off). This register is cleared by cold reset.

29 December 1995 113 \

Polo Software Programming Guide Stratus company Confidential

12.4.5 Quicktime

Type: Read Only in Privileged page 7FF Offset [22:0]: 7FF808

Type: Read Only In non-privileged page 7FE Offset [22:0]: 7FE808

Cold reset clears this register, warm reset has no effect

A read-only view of bits 31 :0 of the master counter. Bit 24 of quicktime increments once per second.

12.4.6 Master Jiffy Counter

Type: Read Write in Privileged page 7FF Offset [22:0]: 7FF810

Read only in non-Privileged page 7FE Offset [22:0]: 7FE810

Cold reset clears this register, warm reset has no effect

31:8 Jiffy Counter/Master Counter

This field contains bits 23:0 of the master 12MHz counter. Bits 31:16 are often re¬ ferred to as the Jiffy Counter. A Jiffy interval is 15.258789 usec, or 2' ¹⁶ seconds.

7:0 Reload Value

This is the counter reload value. The least significant 8 bits of the master counter get reloaded with the contents of this register when they roll over. The reload value is programmable to allow accurate clock operation when the machine is running with a different speed crystal; i.e. 12MHz is no longer really 12MHz. The reload value should be set to 256 minus the number of 12MHz clock ticks in 15.258789 usec. In systems with a standard clock this works out to 49 hex (73 decimal).

This register should only be written when the counter is stopped (Master Counter Enable bit in the Jiffy Control register is off). This register is cleared by cold reset

12.4.7 Jiffy Control

Type: Read/Set Clear in Privileged page 7FF Offset [22:0]: 7FF818

Read only in non-Privileged page 7FE Offset [22:0]: 7FE818

Cold reset clears this register, warm reset has no effect

31:1 Reserved

0 Master Counter Enable

When this bit is set, the master counter (and thus the Time of Day, Jiffy, and Quick¬ time) increments normally. Clearing this bit stops the counter.

29 December 1995 114

Polo Software - . „grammιng Guide Company Confidential

12.5 PCIB/HSC2 Specific Xbus Register Descriptions

These registers are the PCIB/HSC2 specific Xbus/Golfbus registers that reside in a PCIB/HSC2 board. Registers descπbed here are implemented in the Gambit ASIC on the PCIB board. Refer to section 4.5, PCIB MIO/IOBus Compatible System Address Map, for details on the address mapping.

Table 24. PCIB/HSC2 Specific Xbus/Golfbus Register Map.

Table 25 illustrates the formation of system I/O addresses for the Xbus specific and Xbus common registers found on the PCIB board. Note that the slot number is inverted. Polo PCIB boards do not run duplexed so they will not respond to paired accesses.

Table 25. MIO Compatible Register System Address Formation

a paired space -> - 1 , unpaired space -> - 0. must be non-paired

Below is a register by register description of the Gambit/Mirage specific Xbus/Golfbus registers. Note that registers are restricted to 32 bits and are spaced at 32 bit boundaries. The actual register width vanes according to its definition. Reads to any undefined holes in the address space return undefined, but deterministic data. For writes, there is no effect.

The format of this information is either a bit number or a bit encoding and a functional name, depending on the configuration of the register, followed by a description of the function of that bit or bit encoding.

All registers are reset to zero unless otherwise noted.

12.5.1 Gambit Maintenance Attention Request [31:0] Type: Read/Set/Clear- C/D different Offset [22:0]: 7FF080 Cold reset clears this register. Warm reset and going through sync point have no effect

This CD different register indicates which PCI slot (including the Gambit and Mirage host bridges) caused a maintenance interrupt. Note: In Polo, this is a C/D different register, but it differs from other C/D different registers that bits 8, 9, 10, 11 and 12 are cleared by writing

29 December 1995 115

- °r^ :>

Polo Software Programming Guide Stratum company Confidential

32*00000008, 32'h00000009, 32'h0000000A,32'h0000000B and 32'h0000000C respectively.

31,15 Reserved

30,14 Reserved

29,13 Reserved

28,12 Host_bridge (D-side)

27,11 PCI slot 3

26,10 PCI Slot 2

24,8 PCI Slot 0

23,7 Reserved

22,6 Reserved

21,5 Reserved

20,4 Host_bridge (C-side) - Polo Only

19,3 PCI slot 7 - Polo Only

18,2 PCI slot 6 - Polo Only

17,1 PCI slot 5 - Polo Only

16,0 PCI slot 4 - Polo Only

12.5.2 lOBus Status

Type: Read/Clear Offset [22:0]: 7FF008

Cold reset clears this register.

Warm reset and going through sync point have no effect.

31:22 Reserved

21 Xbus/Golfbus Mainteneπace Interrupt asserted due to PCIB/HSC2 Attention Request. See the SAM Maintenance Attention Request Register tor details.

20:0 Reserved

29 December 1995 _ , 116

Polo Software Programming Guide ^at s Company Confidential

12.6 SAM Interface Register Descriptions

These registers are the I/O registers that reside on a SAM board from a software perspective. For Polo, these registers exist either in the Cyclops ASIC on the CPU board or in the Gambit ASIC on the PCIB board. For HSC2, these registers exist in the Mirage ASIC located on the HSC2 board. Refer to section 4.5, PCIB MIO/IOBus Compatible System Address Map, for details on the address mapping.

The SAM compatible ASIC's register space is divided as follows. The 16k space is divided into four 4k pages to simplify the decode of this region in the Cyclops Polo ASIC and Gambit Polo ASIC.

Polo implements a single Map RAM in the CPU, four sets of per bus registers (one set for each of four PCI buses) and 16 sets of per slot registers (one set per each of 16 possible PCI slots). HSC2 implements a single Map RAM in the Mirage ASIC, one set of per bus registers and 4 sets of per slot registers (one set per each of 4 possible PCI slots).

All accesses alias to the appropriate physical register, although software locking is required to prevent conflicts. See section 5.5 for software lock restrictions on these registers.

Table 26. SAM Compatible ASIC Register Map

Note that registers are restricted to 32 bits and are spaced at 64 bit boundaries. The actual register width vanes according to its definition. The byte significance of these registers is big- endian. Reads to any undefined holes in the address space return zeros. For writes, there is no effect The following table illustrates the formation of a SAM compatible I/O register space address on the system bus.

All SAM registers are accessed through Xbus slots 2 and 3 in Polo systems. Accesses to registers associated with PCI slot numbers greater than seven will not be responded to in Polo systems.

Table 27. SAM Compatible Register System Address Formation

a. paired space -> - 1; unpaired space -> - 0 b. page offset from table 26

A bit by bit description follows (bit 31 being most significant and bit zero being the least significant bits). The format of this information is either a bit number or a bit encoding, depending on the

29 December 1995 117

Alt

Polo Software Programming Guide Stratus company Confidential

configuration of the register, and a functional name, followed by a description of the function of that bit (bit encoding). All registers are reset to zero unless otherwise noted.

29 December 1995 118

Polo Software Programming Guide wi atus Company Confidential

12.6.1 SAM Interface Registers - Page 3

These register reside in the Gambit and Mirage ASIC. For this group of registers, there is only a single register for the entire Polo and HSC2 system to which all 16 PCI (Polo) or 4 PCI (HSC2) slot addresses map.

Table 28. SAM Compatible I/O Space Map - Page 3

Note that unlike a Polo/Gambit the C-side and D-side Mirage's bridge a single PCI bus to the system. Therefore a HSC2 system does not have support for multiple disk shelves (SCSI port 0 & 1) and will react to accesses to SCSI Port 1 (analogous to Gambit D-side) only. Accesses to SCSI port 0 (analogous to Gambit C-side) are ignored.

12.6.1.1 Disk LED Control Type: Read/Write Offset [11:0]: FF8

Cold reset as documented below, warm reset has no effect

This register is used to control the amber LEDs on the front of the Polo disks via the Storageworks specified "Disk fault bus." Although this register is read/write, it should be treated as write-only by software. The reason is that the PCIBs in Xbus slots 2 and 3 share the fault bus lines. Writes to this register in either of the PCIBs set the disk LEDs, but only the most recently written PCIB will return a value from this register that is guaranteed to correspond with the actual state of the disk LEDs.

31:6 reserved.

5 Disk shelf select bit. When writing this register and this bit is a 0, SCSI port 0 is selected

(C side Gambit). When writing this register and this bit is a 1 , SCSI port 1 is selected (D side Gambit). This bit has no effect when reading this register.

Note: This bit is not supported on HSC2.

4 LED on bit This bit is set to a 1 to turn on the amber device fault LED for the device whose

SCSI TID is specified by bits 2:0, or all devices when bit 3 is set to a one. Setting this bit to 0 turns off the LED(s).

3 LED address test bit. This bit is used in conjunction with bit 4 to test the amber device fault

LEDs on all disks in the self. Setting bit 3 to a one overrides the TID set by bits 2:0.

2:0 These bits define the SCSI device Target IDentification (TID).

12.6.1.2 Disk Status

Type: Read/Set/Clear Offset [11:0]: FFO

Cold reset as documented below, warm reset has no effect

29 December 1995 119 V>

Polo Software Programming Guide Stratus company Confidential

CD Different (Polo Only): read via unpaired space

This register displays information based on the Storageworks disk subsystem SHELF_OK and SWAP . signals. There are 4 SCSI ports available... C side LOCAL, C side REMOTE^ D side LOCAL and D side REMOTE.

31:15 reserved

30:14 reserved

29:13 reserved

28:12 reserved

27,11 D side REMOTE Swap Interrupt Disable - Setting this bit by writing 0000008Bx suppress¬ es the maintenance interrupt that normally is issued when a REMOTE swap on the D side occurs. Disabling the interrupt does not disable the setting and clearing of the REMOTE Swap has occurred bit. This bit is cleared by cold reset, or by writing OOOOOOOBx.

26,10 D side REMOTE Swap has Occurred. This bit is set when a disk has been inserted or re¬ moved from the D side disk shelf. It stays set until explicitly cleared by writing OOOOOOOAx, or until it is cleared by cold reset. Attempting to set this bit by writing 0000008Ax has no effect. When this bit is set, a maintenance interrupt is issued un¬ less masked by D side REMOTE Swap Interrupt Disable bit.

25,9 D side Swap Interrupt Disable - Setting this bit by writing 00000089x suppresses the main¬ tenance interrupt that normally is issued when a swap on the D side occurs. Dis¬ abling the interrupt does not disable the setting and clearing of the Swap has occurred bit. This bit is cleared by cold reset, or by writing 00000009x.

24,8 D side Swap has Occurred. This bit is set when a disk has been inserted or removed from the D side disk shelf. It stays set until explicitly cleared by writing OOOOOOOSx, or until it is cleared by cold reset Attempting to set this bit by writing 00000088x has no ef¬ fect When this bit is set, a maintenance interrupt is issued untess masked by D side Swap Interrupt Disable bit.

19,3 C side REMOTE Swap Interrupt Disable - Setting this bit by writing 00000083x suppress¬ es the maintenance interrupt that normally is issued when a REMOTE swap on the C side occurs. Disabling the interrupt does not disable the setting and clearing of the REMOTE Swap has occurred bit. This bit is cleared by cold reset, or by writing 00000003X.

This bit is not implemented in a HSC2 system.

18,2 C side REMOTE Swap has Occurred. This bit is set when a disk has been inserted or re¬ moved from the C side disk shelf. It stays set until explicitly cleared by writing 00000002x, or until it is cleared by cold reset Attempting to set this bit by writing 00000082x has no effect. When this bit is set, a maintenance interrupt is issued un¬ less masked by the C side REMOTE Swap Interrupt Disable bit.

This bit is not implemented in a HSC2 system.

17,1 C side Swap Interrupt Disable - Setting this bit by writing 00000081 x suppresses the main¬ tenance interrupt that normally is issued when a swap on the C side occurs. Dis-

29 December 1995 , _Λ) 120

Polo Software F.,jramιnιng Guide _^ ,atus Company Confidential

abling the interrupt does not disable the setting and clearing of the Swap has occurred bit. This bit is cleared by cold reset, or by writing 00000001 x.

This bit is not implemented in a HSC2 system.

16,0 C side Swap has Occurred. This bit is set when a disk has been inserted or removed from the C side disk shelf. It stays set until explicitly cleared by writing OOOOOOOOx, or until it is cleared by cold reset. Attempting to set this bit by writing OOOOOOSOx has no ef¬ fect When this bit is set, a maintenance interrupt is issued unless masked by the C side Swap Interrupt Disable bit.

This bit is not implemented in a HSC2 system.

12.6.1.3 Power Supply LED

Type: Read Write Offset [11 :0]: FE8

Warm and Cold reset have no effect on this register. Polo Only Register

31:3 Reserved.

2 Red LED bit Writing a 1 to this bit position turns the Red LED on. Writing a 0 turns the

Red LED off. This bit will also be set if a local power supply fault is detected.

1 Yellow LED bit Writing a 1 to this bit position turns the Yellow LED on. Writing a 0 turns the Yellow LED off. This bit will also be cleared if a power supply fault is detected on either the local or remote power supplies.

0 Green LED bit Writing a 0 to this bit position turns the Green LED on. Writing a 0 turns the

Green LED off. This bit will also be cleared if a local power supply fault is detected.

12.6.1.4 Power Supply Status(Vanguard powe supply)

Type: Read Only Offset [11 :0]: FEO

Warm and Cold reset have no effect on this register. Polo Only Register

31:10 reserved.

9 Local power supply fault bit. This bit is set if the local power supply reports a fault

8 Remote power supply fault bit. This bit is set if the remote power supply reports a fault.

7:0 Local power supply ID.

12.6.1.5 Fan Speed Control

Type: Write Only Offset [11:0]: FD8

Cold reset clears this register. Polo Only Register

This register is used to control the fan speed. On power on, this register is cleared, there by forcing the fan to run at normal speed. Writing oxhβo to this register sets this register, there by forcing the fan to run at higher speed. Writing oxhOO to this register clears this register, there by forcing the fan to run at normal speed.

29 December 1995 _ 121

9781

Polo Software Programming Guide Stratub company Confidential

12.6.2 SAM Interface Registers Page 2

These registers reside in the Cyclops ASIC in Polo systems, however they are addressed through Xbus slot 2 or 3. In a HSC2 system, these registers reside in the Mirage ASIC and should be addressed through the normal Golbus slot.

In a Polo system, there is only a single set of registers to which all 16/4 PCI slot addresses map.

In a HSC2 system, there is a set of registers tor each PCI bus (Dune).

Table 29. SAM Compatible I/O Space Map - Page 2

12.6.2.1 Address Table Command and IOVA Register

Type: Read/Write Offset [11:0]: FF8

Cold reset clears this register, warm reset has no effect.

This register, combined with the three Address Table Data registers, is used to access the map rams for IOVA translation. See section 5. for a complete description of the address mapping function. To write the map rams, first load up the Address Table Date registers with the desired wnte value then write the Address Table Command and IOVA Register with the proper device index and page and bit 31 equal to 1. In order to read a map ram simply wnte to this register with the proper device index and page and bit 31 set to zero. Follow this initial write with a read of any ot the four registers to retneve the data.

12.6.2.2 Address Table Data 2 Register Type: Read/Write Offset [11:0]: FFO Cold reset clears this register, warm reset has no effect.

Refer to section 12.6.2.1 , Address Table Command and IOVA Register, tor a descπption of this register.

31 - 30 Reserved

29 December 1995 122

Polo Software Programming Guide όuatus Company Confidential

29 -26 PCI Slot Number

Note: In HSC2, the most significant bit of the PCI slot number (bit [29]) is not used due to space limitations in the Mirage Map RAM. Since a HSC2 only supports 4 PCI devices this should not be considered an issue.

25-5 Reserved

4 Option bit - Xbus/Golfbus incoherent Memory Access

3 Option bit - Swap 16 or 32 bit Endian

2 Option bit - Xbus/Golfbus Lock Cycle

1 Option bit - Date Pre-read

0 Entry valid bit

12.6.2.3 Address Table Data 1 Register

Type: ReadWrite Offset [11 :0]: FE8

Cold reset clears this register, warm reset has no effect.

Refer to section 12.6.2.1 , Address Table Command and IOVA Register, for a description of this register.

31 :16 Block Xbus/Golfbus Virtual Index

15:12 Reserved

11:2 Block Xbus/Golfbus Ending Physical Address

1:0 Reserved

12.6.2.4 Address Table Data 0 Register

Type: Read Write Offset [11 :0]: FEO

Cold reset clears this register, warm reset has no effect.

Refer to section 12.6.2.1 , Address Table Command and IOVA Register, for a description of this register.

31-2 Block Xbus/Golfbus Starting Physical Address 31-12 Physical Cache Tag 11-2 Line Offset

1:0 Reserved.

29 December 1995 123

/US97/09781

Polo Software Programming Guide Stratus company Confidential

12.6.3 SAM Interface Registers- Page 1

Each Gambit/Mirage ASIC has one of each of these registers for the one PCI bus (up to four controllers per bus) it controls.

Table 30. SAM Compatible Common IO Space Map Page 1

12.6.3.1 PCI config_addr Type: Read/Write Offset [11:0]: FF8

Cold reset clears this register, warm reset has no effect,

IO reads and writes to this register cause no PCI cycles. IO reads and writes to the 'PCI config_data register' use the information in the 'PCI config_addr register' to perform a configuration cycle on the PCI bus except for device zero i.e. ttie bridge which is accessed internal to Gambit/Mirage using this same mechanism. Thus configuration reads and writes to the host bridge's configuration space will not cause PCI cycles but will instead be handled internal to the host bridge.

Special cycles will be generated when the 'Bus Number' equals zero i.e. the bridges secondary PCI bus, and the 'Device Number' is all ones, and the 'Function Number' is all ones and the 'Register Number' is all zeros.

31 Enable config_date - As per the PCI spec config_data is disabled if this bit is a 0.

30-24 Reserved - read only return zeros

23-16 Bus Number - encoded value to select 1 Of 256 PCI buses in a system.

15-11 Device Number - encoded value to select 1 of 32 device on a given bus.

10-8 Function Number - encoded value to select 1 of 8 functions on a specific device.

7-2 PCI Configuration Register DWORD Index - specific bytes trom BEs.

1-0 2'bOO - Read only

The configuration type will be added in by hardware based on the 'Bus Number' - type 00 config to attached bus. type 01 config passed to 'Bus Number ^* . Refer to the PCI Local Bus Specification for more information on type 00 and type 01 commands.

29 December 1995 124

Polo Software Programming Guide Sυαtus Company Confidential

12.6.3.2 PCI conflg_data

Type: ReadWrite Offset [11 :0]: 7F0

Cold reset clears this register, warm reset has no effect.

10 Reads and Writes to this register result in PCI configuration cycles on the PCI bus as defined by the address in 'conftg_addr' except for device zero i.e. the bridge configuration space is accessed internal to Gambit Mirage using this same mechanism. The PCI configuration date is little-endian. The Xbus and Golfbus are big-endian. Reads and writes to the PCI configuration space of either the host bridge or the mated PCI adapter are always byte-swapped to provide 'byte address consistency' rather than 'MSByte consistency'. Where significance of bytes is rearranged by the swapping mechanism, software will have to re-order to restore significance. Refer to section 5.6 for a discussion of byte ordering (big endian versus little endian) on configuration cycles.

Setting bit 31 in the 'PCI config_addr register' will cause accesses to the this register to be enabled, tf bit 31 of 'PCI config_addr register' is 0 writes will be ignored, and reads will return O's and a failed.op status.

31-0 Configuration data

12.6.3.3 Test Control

Type: Read/Set/Clear Offset [11 :0]: FE8

Cold reset clear this register.

31:16 Reserved

15 Disable Protocol error signal generation

This bit is set by software to disable Protocol error signal generated by PCI core logic.

When this bit is set, the PCI core logic does not assert protocol_erτor signal to rest of

Gambit/Mirage logic. When this bit is set, bit 5 of Polo PCI Error register will not be set. This bit is cleared by cold reset, or by writing OOOOOOOFx. It is set by writing

OOOOOOβFx.

14 Disable Peer to Peer error signal generation

This bit is set by software to disable Peer-to-Peer error signal generated by PCI core logic. When this bit is set, the PCI core logic does not assert peer_to_peer_error signal to rest of Gambit/Mirage logic. When this bit is set, bit 6 of Polo PCI Error register will not be set. This bit is cleared by cold reset, or by writing OOOOOOOEx. It is set by writ¬ ing OOOOOOΘEx.

13 PCI read return ahead enable (Polo Only)

This bit is implemented in Rev. 1 Gambit only. When this bit is set, the Gambit sends read return data to the PCI bus as soon as data is available on post.qual bus. This will im¬ prove the PCI bus read performance for cade line read. If this bit set to to zero, the Gambit waits till last peace of data is available on post_qual bus for cache line read before it sends data to the PCI core. This bit is cleared by cold reset, or by writing OOOOOOODx. It is set by writing OOOOOOδDx.

12 Disable PCI drive/check error signal generation

This bit is set by software to disable PCI drive check error signal generated by PCI core logic. When this bit is set, the PCI core logic does not assert drive_check_error signal to rest of Gambit/Mirage logic. When this bit is set, bit 9 of the Polo PCI Error register will not be set This bit is cleared by cold reset, or by writing OOOOOOOCx. It is set by writing 0000008Cx.

11 Disable System error signal generation

29 December 1995 125

Polo Software Programming Guide Stratus υompany Confidential

This bit is set by software to disable system error signal generated by PCI core logic. When this bit is set, the PCI core logic does not assert system_error signal to rest of Gambit/Mirage logic. When this bit ts set, bit 28 of Polo PCI Error register will not be set. This bit is cleared by cold reset, or by writing OOOOOOOBx. It is set by writing 0000008Bx.

10 Disable Parity error signal generation

This bit is set by software to disable panty error signal generated by PCI core logic. When this bit is set, the PCI core logic does not assert panty_error signal to rest of the

Gambit/Mirage logic. When this bit is set, bit 27 of Polo PCI Error register will not be setThis bit is cleared by cold reset, or by wnting OOOOOOOAx. It is set by wnting

000 008AX.

9 Reserved.

8 Flush Mirage Write Buffer Map Entry Caches (HSC2 Only)

Wnting to set this bit will invalidate the two wnte buffer entry caches. This will insure that Mirage will go to the Map RAM for the next PCI write access and allows software the ability to gurantee consistent data. Refer to the Mirage Specification for more details.

Writing to clear this bit has no affect. This bit will always return a zero when read.

7 Inhibit Broken SAM on SERR

This bit is set by software to inhibit PCI error logic from removing a SAM from service when an error is intentionally caused for diagnostic testing purposes.This bit is cleared by cold reset, or by wnting 00000007x. It is set by writing 00000087x.

6 Error Data Recorded/Rearm Error and Broken Registers

This bit is set by hardware when an error condition that requires logging in any of the error registers occurs. Writing a 0 to this bit clears the error state of the chip and "re¬ arms" them so they will capture the state of the next applicable error condition.This bit is cleared by cold reset, or by writing 00000006x. This bit can not be set It is set by wnting 00000086x.

5 Force PCI data parity error

This bit is set by software to intentionally cause a PCI date panty error for diagnostic test¬ ing purposes.This bit is cleared by cold reset, or by writing 00000005x. It is set by writing 00000085x.

4 Force PCI address panty error

This bit ts set by software to intentionally cause a PCI address panty error for diagnostic testing purposes.This bit ts cleared by cold reset, or by wnting 00000004x. It is set by writing 00000084X

3:0 Reserved

12.6.3.4 PCI Error

Type: Read Only Offset 111:0]: FEO

Cold reset clears this register, warm reset has no effect. IO Reads and Wntes to this register cause no PCI cycles.

This register is valid if Status[l6] is set (PCI Error detected).

31-29 PCI Grant Slot#(3-0) indicating mastership during registered error. This information is en¬ coded in the following manner.

1 5 . 126

Polo Software F . - jrarrn ..ing Guide _ .. atus Company Confidential

0: Slot 0 1 : Slot 1 2: Slot 2 3: Slot 3 4: Host

28 Bridge detected SERR# on PCI Bus. 27 Bridge detected PERR# on PCI Bus

26 Bridge detected Master Initiated Termination

25 Bridge detected Target Initiated Termination

24 Reserved.

24 Host request FI FO timeout error.

Note: When this error occurs, other info in this register may or may not repre¬ sent correct info about PCI bus transaction that caused this error. Refer to sec¬ tion 9.6.2.3 for details.

23:20 Reserved.

19-16 PCI Command.

15-12 Reserved

11 IDLE timeout after GNT#

PCI master failed to start an access within 16 clocks after GNT# asserted. Note that the PCI command field and IOVA values are not applicable.

10 Time Out Error (see PCI Common Logic Spec.)

9 Drive Check Error (see PCI Common Logic Spec.)

8 Retry Count Error (see PCI Common Logic Spec.)

7 Disconnect Count Error (see PCI Common Logic Spec.)

6 Peer to Peer Error (see PCI Common Logic Spec.)

5 Protocol Error (see PCI Common Logic Spec.)

4-0 Reserved

12.6.3.5 PCI IOVA Error

Type: Read Only Offset [11:0]: FD8

Cold reset clears this register, warm reset has no effect IO Reads and Writes to this register do not cause PCI cycles.

This register is valid if Status[16] is set (PCI Error detected).

31-0 Xbus/Gbus IOVA for the PCI transaction which got an error.

12.6.3.6 SAM Status

Type: Read only Offset [11:0]: FD0

Cold reset clears this register, warm reset has no effect.

State Reduction (Off-line, Not-Ready):

29 December 1995 _^ 127

Polo Software Programming Guide Stratus Confidential

The contents of the state reduction, are frozen when the SAM first goes off-line or broken, so that the comparators that caused the state reduction are flagged. This is so that comparators that miscompare further down ttie line (after the error has rippled through the logic) do not obscure the cause of the onginal failure.

Writing to clear any of the state reduction bits will have no effect. The state reduction bits are re¬ armed via the Rearm Error register bit in the Test Control register.

31:17 Reserved.

16 SAM Maintenance Interrupt issued due to PCI Error/Abort condition.

Refer to the PCI Error register and the PCI Configuration Space Status registers for details.

15:14 Reserved

13 Software Set Off-line Command.

This bit is set by writing 00000081 x to the page 0 Bus Interface State register.

12 Map Error (HSC2 Only)

This bit is set when an I/O Address Map Error is detected. Refer to the CPU Specific Xbus Register Descriptions on page 112 for further details about the error.

11 Failed Op (HSC2 Only)

When set, this bit indicates that the return for a transaction has timed-out and zeros were returned to prevent the device from hanging. Status[10:9] indicate which slot (Sam) was responsible for issuetng the read request. Note that this Sam's stete will have changed from online to online - not ready.

10-0 Reseved

12.6.3.7 Arbitration Freeze Count Max. Value

Type: Read/Write Offset [11:0]: FBO

Cold reset clears this register, warm reset has no effect on.

After certain types of PCI errors, arbitration logic is "frozen" and a PCI_reset is issued to the affected PCI slot. Writing an 8 bit value to this register determines how many 24Mhz clocks the arbitration logic will remain "frozen" after PCI_reset is de-asserted.

12.6.3.8 Host Request FIFO Timeout Value Register

Type: ReadWrtte Offset [11 :0]: FA8

Cold reset sets this register to 32' h0100.0000; warm reset has no effect.

This register contains Timeout value tor host request FIFO. This register is initialized to 32'h0l00_00 on power on. After a host request for PCI bus access is sent, if the request is not completed by number of 12MHz clock ticks specified in this register, the PCIB or HSC2 is broken and host request FIFO timeout error bit in the PCI error register is set. Setting this register to 32'h0000_0000 disables the host request FIFO timeout error.

29 December 1995 128

Polo Software r lOgrai.iming Guide stratus Company Confidential

12.6.4 SAM Interface Registers - Page 0

There is a copy of each of these registers for each slot that a given ASIC controls. Thus the Gambit and Mirage have four copies of each register.

Table 31. SAM Compatible I/O Space Map - Page 0

12.6.4.1 Board Reset

Type: Read/Set Clear Offset [11:0]: FB8

Cold and warm reset affect this register as documented below

This register behaves almost identically to the Xbus Board Reset register (or Golfbus Board Reset register) listed in section 12.3.3. except that PCI reset is also activated through this register. PCI reset is asserted by an explicit PCI reset access to this register, or by a power-up reset. A software initiated cold or warm reset also asserts PCI reset. NOTE: A PCIB hardware or software cold reset (as indicated in the Xbus Board Reset register) has no effect on this register.

31:7 These read bitswrite bits are reserved for additional board specific resets.

6 Clear SAM entry cache valid bit (HSC2 only)

Writing to set this bit generates a pulse which will invalidate the entry cache in the reader and flush any read data in the read RAM. This will insure that the reader will go to the Map RAM for the next PCI access and allows software the ability to guaran¬ tee consistent date when using the pre-read option.

Note that the wife buffer caches are unaffected by this operation. Write buffer caches can be invalidated through the Test Control Register (page 1 ).

Writing to clear this bit has no effect. Reading this bit will always return a 0

Software Note: Clearing the entry cache during a PCI initiated read transaction can cause unpre¬ dictable results.

5 Software initiated PCI Reset

These read bits/write bits are reserved for additional board specific resets.hen read-

29 December 1995 129

.Λϊ.

Polo Software Programming Guide Stratus Company Confidential

ing this register, this bit =1 indicates that the board has received a software generat¬ ed PCI reset since this bit was last cleared. Writing a 00000085x to this register will freeze PCI arbitration logic, generate a PCI reset pulse on the board and set this bit. Writing OOOOOOOδx to this register clears this bit. The activation of either of the cold reset bits also clears this bit. For SAM, PCI Reset clears the internal PCI macro, in¬ cluding the configuration space and state machines. It also causes the assertion of a PCI reset to the attached PCI adapter

Software Note: On Mirage, PCI resets also cause the the Reader hardware to reset simitar to a warm_reset. Software initiated PCI resets during a PCI initiated read transaction can cause unpredictable results.

4 Software initiated Warm Reset

When reading this register, this bit -1 indicates that the board has received a soft¬ ware generated Warm Reset since this bit was last cleared. Writing a 00000084x to this register generates a Warm Reset reset pulse on the board and sets this bit. Writ¬ ing 00000004x to this register clears this bit The activation of either of the cold reset bits also clears this bit For SAM, Warm Reset Clears all bus state machines, includ¬ ing PCI state machines, clears all non-persistent error conditions and causes a PCI Reset.. All pending bus activities are cleared. The effects of Warm Reset on individu¬ al registers is specified in the heading of each register.

3:2 Reserved

1 Software initiated Cold Reset

When reading this register, this bit =1 indicates that the board has received a soft¬ ware generated Cold Reset since this bit was last cleared. Writing a 00000081 x to this register generates a Cold Reset reset pulse on the board and sets this bit Writ¬ ing 00000001 x to this register clears this bit The activation of the Power-up cold re¬ set bit also clears this bit. For SAM, a Software Cold Reset clears all error reporting registers in addition to clearing everything effected by a Warm Reset and PCI Reset. The effects of Cold Reset on individual registers is specified in the heading of each register.

0 Cold Reset due to Power up

When reading this register, this bit = 1 indicates that the board has received a power- up generated Cold Reset since this bit was last cleared. This bit is cleared by writing a 000000O0X to this register or by a software initiated Cold Reset. When this bit is set by a power-up reset ell other bits in this register are cleared. This bit is NOT cleared by Warm Reset.

12.6.4.2 LED Control

Type: Read Write Offset [11:0]: FB0

Cold and warm reset affect this register as documented below

Provided there is a PCI card present in the specified slot, this register functions identically to the register described in section 12.3.3. with the addition that PCI reset will set the red LED and clear the yellow and green LEDs. Refer to section 12.6.4.1 for more information on PCI reset. If no PCI card is present all LEDs will remain off.

12.6.4.3 SAM Host Interrupt Bit

Type: Read/Write Offset [11:0]: FA0

Cold reset clears this register. Warm reset has no effect.

Bits 4:0 of this register points to the bit to be written in the host interrupt register, EIR and the least

29 December 1995 _ 130

Polo Software r- .υgraniining Guide _ -atus Company Confidential

significant byte is also the pattem to be written to the table in host memory.

31:6 Reserved.

5 1 , this is the "set" bit for the read/set/clear register

4:0 Point to the bit location to be set in the host interrupt register.

12.6.4.4 SAM Interrupt Mask

Type: Read/Write Offset [11:0]: F98

Cold reset clears this register. Warm reset has no effect.

These bits are used to individually mask or enable the four interrupt source signals, INTA# - INTD# All four interrupt lines are tied together in hardware and must be masked/unmasked as a group.

31:1 Reserved.

0 interrupt mask - (1 -enabled,

12.6.4.5 SAM Interrupt Source

Type: Read Only Offset [11:0]: F90

Cold reset clears this register. Warm reset has no effect.

This bit is used to indicate if any of the four interrupt source signals, INTA# - INTD#, are active with interrupts pending. Most single function PCI devices only use INTA#. All four interrupt lines are tied together in hardware and can only be checked as a group.

31:1 Reserved.

0 interrupt status - (1 -active, 0-inactive)

12.6.4.6 SAM Host Interrupt Address Pointer

Type: Read/Write Offset [11:0]: F88

Cold reset clears this register. Warm reset has no effect.

This register contains the IO address of the EIR register in one or all cougars. 31 :2 Bits 31 :2 point to the location of the interrupt register in host memory. 1:0 Reserved.

12.6.4.7 SAM Host Interrupt Table Pointer #2

Type: Read/Write Offset [11:0]: F80

Cold reset clears this register. Warm reset has no effect

These bits are the upper 16 bits of the address of the interrupt table entry in host memory.

31:16 Reserved.

15:0 Bits 47:32 of the table base address pointer.

12.6.4.8 SAM Host Interrupt Table Pointer #1

Type: Read Write Offset [11:0]: F78

Cold reset clears this register. Warm reset has no effect.

29 December 1995 131

09781

Polo Software Programming Guide Stratus ompany Confidential

These bits are the lower 32 bits of the address of the interrupt table entry in host memory. 31:0 Bits 31 :0 of the table base address pointer.

12.6.4.9 SAM Configuration

Type: Read/Write Offset [11:0]: F70

Cold reset clears this register, warm reset has no effect.

31-0 Reserved.

12.6.4.10 PCI IO Space Offset

Type: Read/Write Offset [11 :0]: F68

Cold reset clears this register, warm reset has no effect.

IO reads and writes to this register cause no PCI cycles. IO reads and writes to the 16KB window known as the SAM PCI/IO space window use the information in the PCI tO Offset register to map the base address of the 6KB IO space window. Refer to section section 4.5.1 for further information on the use of this register. This register is big endian.

31-14 Upper 18 bits of 32-bit PCI address for accesses to the 16KB PCI IO space window. PCI only defines use of the first 64KB of IO space, but the interface supports IO space ac¬ cesses over the entire 32-bit address.

13-0 Reserved

12.6.4.11 PCI Memory Offset

Type: Read Write Offset [11:0]: F60

Cold reset clears this register, warm reset has no effect.

IO reads and writes to this register cause no PCI cycles. Memory reads and writes to the 64KB window known as the SAM Memory space window use the information in the PCI Memory Offset register to map the base address of the 64KB memory window. Refer to section 4.5.1 for further information on the use of this register. This register is big-endian.

31-16 Upper 16 bits of 32-bit PCI address for accesses to the 64KB PCI memory window. A 64KB window to memory can be put anywhere in the 32-bit PCI memory range. The PCI base address register in the configuration register space of the PCI adapter must be set to align with this window.

15-0 Reserved

12.6.4.12 Bus Interface State

Type: Read/Set Clear Offset [11:0]: F58

Cold reset clears this register; warm reset effect is documented betow. All bits remain cleared if there is not a PCI card in the specified slot.

These bits reflect the current state of the SAM board.

31:3 Reserved

2 Off-Line - Not Ready

Board capable of receiving and responding to Xbus Golfbus requests, but is incapa¬ ble of performing Xbus/Golfbus initiated PCI requests. There is non-zero error state in the Gambit.

29 December 1995 _r 132

Polo Software . rogr«. .iming Guide Jtratus Company Confidential

This status bit is used to indicate the occurrence of a non Xbus/Golfbus related hard¬ ware fault which was detected by the checking logic. This bit is set automatically by hardware. Writing a 00000082x to set, or a 00000002x to clear will have no effect. This bit is cleared by a cold reset, or by writing the Test control register with a Rearm Error Command.

Off-Line - Ready

Board capable of receiving and responding to Xbus/Golfbus requests but is incapable of performing Xbus initiated PCI requests. No error conditions are present.

This state is entered automatically after a cold or warm reset (provided there are no errors detected by the Gambit or Mirage ) or when the error state of the chip is cleared via a Rearm Error Command. Writing a 0000081 x to set, causes SAM Sta- tus[l 3] to get set and a Attention Maintenance Interrupt to be generated. Clearing this bit has no effect. (Note if the SAM is broken you won't be able to read this.)

On-line

Board capable of receiving and responding to Xbus/Golfbus requests. Board capable of performing Xbus initiated PCI requests. PCI adapter enabled to post requests.

Writing a OOOOOOSOx to set, bit causes the PCI arbiter to be enabled provided there is a non-zero error state. Writing a OOOOOOOOx to clear causes the Gambit Mirage to go Off-line - Ready but a Attention Maintenance Interrupt is not generated. Note if the SAM is broken you won't be able to read this.)

29 December 1995 133

rhis document contains confidential and proprietary information of Stratus Computer, nc, and any reproduction, disclosure, or use in whole or in part is expressly xohibited, except as may be specifically authorized by prior written agreement or jermission of Stratus.

Xbus Functional Specification

JED-00155

"If You Want Fault Tolerance, Buy A Jetta"

Will Leavitt (editor)

Conrad Clemson

Jeffrey Somers

John Chaves

David Barbera

21 July 1995

Revision: 1.3

(c) S tratus Computer , Inc.

Appendix II

- S Z-

Xbus Functional Specification Stratus company Confidential

Table of Contents

1. Introduction 8

1.1 Applicable Documents 8

1.1.1 Stratus Documents 8

1.1.2 Other Documents 8

1.2 Revision History 9

2. Functional Overview 10

2.1 Introduction 10

22 Architectural Altematives 11

2.2.1 New Bus 11

2.2.2 PCI Based Bus 11

2.2.3 Ibus based Bus 12

2.2.4 Golfbus Based Bus 12

2.3 Golfbus Logical Overview 12

2.4 Golfbus Physical Summary 13

2.5 Polo XBus Logical Overview 14

2.6 Polo XBus Physical Overview 14

3. Detailed Functional Description 15

3.1 Bus Naming Convention 15

3.1.1 Data Bus Naming Convention 15

3.1.2 Control Bus Naming Convention 16

3.2 Bit Numbering 17

3.3 Terminology 17

4. Xbus Signal Description 21

4.1 Signal Description 21

4.2 The Info Bus 21

4.3 The Control Bus 22

4.4 The Voted Signals 25

4.5 Other Control Signals 27

5. Xbus I/O Interface Registers 28

6. Xbus Protocol 29

6.1 Overview 29

6.2 Bus Operation 29

6.3 Bus Busies 30

6.4 Bus Errors 30

6.4.1 Bus Error Broken Conditions 32

6.4.2 Xbus Fault Analysis 33

6.4.3 Fault Conditions 34

6.4.3.1 CPU Board Faulty Input Circuit - CPU Driving 34

6.4.3.2 CPU Board Faulty Input Circuit - I/O Board Driving 34

6.4.3.3 CPU Board Different Data C-Side and D-side 34

6.4.3.4 CPU Board Faulty Output Circuit - Buffer to Pad Fault 34

6.4.3.5 CPU Board Open - CPU Board Driving 35

6.4.3.6 CPU Board Open - I/O Board Driving 35

6.4.3.7 CPU Board Short 35

21 July 1995 _^ Λ ^

Xbus Functional _μec . -tion atus Company Confidential

6.4.3.8 Backplane Open Etch 36

6.4.3.9 Backplane Short 36

6.4.3.10 I/O Board Faulty Input Circuit 36

6.4.3.11 I/O Board Output Circuit Fault - Buffer to Pad 36

6.4.3.12 I/O Single-side Access - ASIC Parity Gen. Fault 36

6.4.3.13 I/O Single-side Access - ASIC PCI Data Path Fault 37

6.4.3.14 I/O Non-singie-side Access, Different C-D Data 37

6.4.3.15 I/O Board Open 37

6.4.3.16 I/O Board Short 37

6.4.3.17 Transient Fault 38

6.4.3.18 Slow Driver Fault 38

6.4.3.19 Board Not Broken Generation 38

6.4.3.20 Arbitrary Breaking 38

6.5 Summary Bus State Diagram 39

6.6 Arbitration 40

6.6.1 Bus Mastership Upon Bus Grant 43

6.6.2 Special Arbitration Rules for Non-Paired Read Returns 43

6.7 Block Transfers 45

6.7.1 Bus Errors 45

6.7.2 Global Writes 48

6.7.3 Board Breaking 48

6.7.4 Bus Busys 49

6.8 Peer to Peer transactions 49

6.8.1 Bus Enors 51

6.9 Major Transactions 53

6.9.1 Write Transactions 53

6.9.2 Read Transactions 54

6.9.3 Flush and Purge Transactions 56

6.9.4 Load/Clear Transactions 58

6.9.5 Ping Transaction 59

6.9.5.1 The Need for PINGs 59

6.9.52 Detailed Rules During PING Operations 59

6.9.6 busy, ack and error combinations 61

6.10 Xbus Signal Formats 62

6.10.1 trid[6:0] and func_op_ Bus Format 62

6.10.2 Info Bus Format 63

6.10.2.1 Function Code (re and byte enable) Transfer 65

6.10.2.2 Remote/Coherent bits (rc[1:0]) 69

6.10 .3 Byte Enables (byte_en{3:0]) 69

6.11 Board Synchronization/Board States 70

6.11.1 CPU/Memory Bus Interface Modes 71

6.11.2 Simplexed I/O Board Bus Interface Modes 73

6.11.3 Loopback mode 75

6.12 Board Breaking/Board Removal 75

6.12.1 Un-breaking a Board 75

7. Xbus Routing and Xbus Interface Clocking 76

7.1 On-board Clock Generation 76

7.1.1 Clock Phases 76

12 Signal Routing 77

7.3 Clock Fault Tolerance Issues, Outbound Signals 78

8. Board Insertion and Removal 79

21 July 1995 -

Xbus Functional Specification Stratus company Confidential

8.1 Hot Plugging 79

&2 Xbus Interface Testing at Board Insertion 79

9. Xbus Fault Tolerance 80

9.1 Info Bus Protection 80

9.1.1 Parity 80

9.12 Loopback checking 82

9.1.2.1 Loopback on CD different accesses 83

9.122 Loopback on Single-side accesses 83

92 Control Bus Protection 84

9.3 Three-way Voted Signal Protection 84

9.4 Error Reporting 84

9.4.1 Producing Errors 85

9.4.2 Maintenance Interrupts 87

9.42.1 Broken Events 87

9.4.22 Error Events 87

9.4.2.3 Software Control 88

9.4.2.4 Determining the Source of a Maintenance Interrupt 88

9.5 Board Breaking Timing 88

9.5.1 Board breaking and information latching 88

9.52 Board Breaking Timing on Info Loopback Error 90

9.5.3 Board Breaking Timing on Heuristic or Arbitrary Broken 90

10. Reset 91

10.1 Xbus Resets 91

10.2 Power System Generated Reset 92

10.3 Fault Tolerant Issues: 95

10.4 ASIC Pins Required For Reset Functions 95

10.5 DUMB FET'S 95

11. Xbus Physical Partitioning 96

11.1 Into Bus Partitioning 96

11.2 Control Signal Partitioning 96

11.3 Xbus Voted Signal Partitioning 96

11.3.1 ID PROM Partitioning 98

12. Xbus Interface Block Diagrams 100

12.1 Xbus Top Level Block Diagram 100

12.1.1 Port List 100

12.12 Functional Description 100

122 Inbound Pipe Section 101

12.2.1 Port List 103

122.2 Functional Description 108

12.3 PING Section 110

12.3.1 Port List 111

12.32 Functional Description 113

12.4 Xbus Outbound Data and Control Unit 114

12.4.1 Port List 115

12.4.2 Functional Description 118

12.5 Arbitration Logic 126

12.5.1 Port List 127

12.5.2 Functional Description 127

21 July 1995 4

Xbus Functions. ,atus Company Confidential

12.6 IO Register Section 130

12.6.1 Port List 131

12.6.2 Functional Description 135

12.7 Error Checking and Registering Logic 136

12.7.1 Port List 137

12.7.2 Functional Description 149

13. Xbus/Golfbus Differences 151

21 July 1995

Xbus Functional Specification Stratus company Confidential

List of Figures

Figure 1. Xbus interconnect 11

Figure 2. Xbus data Buses 15

Figure 3. Xbus Control Buses 16

Figure 4. Xbus Bit Numbering Convention 17

Figure 5. Terminology - Simple Word Read Transaction 18

Figure 6. Terminology - Simple Line Write Transaction 19

Figure 7. Basic Xbus Cycle 29

Figure 8. Basic Bus Busy Operation 30

Figure 9. Basic Bus Error Operation 31

Figure 10. Bus Error Operation Flow Chart 32

Figure 11. Xbus Interconnect 33

Figure 12. BASIC Bus Operation Flow Chart 39

Figure 13. Inter-CPU Bus Arbitration Network 40

Figure 14. Arbitration After Being Busied 43

Figure 15. High Level Block Transfer 45

Figure 16. Error During First Operation of a Quad Block Transfer 46

Figure 17. Error During Second Operation ot a Quad Block Transfer 46

Figure 18. Error During Third Operation of a Quad Block Transfer 47

Figure 19. Error During Fourth Operation of a Quad Block Transfer 47

Figure 20. Error causing entire block transfer to be canceled 48

Figure 21. Peer to Peer Cycle Initiated by a Simplexed CPU 51

Figure 22. Peer to Peer Non-paired Read from a Duplexed CPU 51

Figure 23. Store and Forward with Error on 2nd Operation of Quad Block Transfer 52

Figure 24. 32 Bit Write Transaction 54

Figure 25. 256 Bit Block Write Transaction 54

Figure 26. 32 Bit Read Transaction 55

Figure 27. Slave Response to a 32 Bit Read 55

Figure 28. 256 Bit Read Transaction 56

Figure 29. Slave Response to a 256 Bit Read Transaction 56

Figure 30. Flush with Modified Data Transaction 57

Figure 31. Slave Response to a Flush with Modified Data Transaction 57

Figure 32. Flush (no modified data) or Purge Transaction 58

Figure 33. Slave Response to a Flush (no modified data) or Purge Transaction 58

Figure 34. PING Transaction 60

Figure 35. PING Transactions, Boundary Condition 1 60

Figure 36. PING Transactions, Boundary Condition 2 61

Figure 37. trid[6:0] and func_op_ Format for the 1 st Cycle of an Info Phase 62

Figure 38. trid[6:4] Format for a PCIB for the 1 st Cycle of an Info Phase 63

Figure 39. trid[6:0] and func_op_ Format for the 2nd cycle of an Info phase 63

Figure 40. lnfo[31 :0] Format of CPU Address FUNC_OP 64

Figure 41. info[31 :0] Format of IOVA Address Bus FUNC_OP 64

Figure 42. lnfo[31 :0] Format During Data Transfers 65

Figure 43. byte_en[3:0] definition 70

Figure 44. CPU/Memory Board state transitions 73

21 July 1995 °y

Xbus Function... opecmcation .ratus Company Confidential

Figure 45. Simplex I/O Board State Transitions 75

Figure 46. Routing of info lines 77

Figure 47. Routing of 3way voted lines 78

Figure 48. Self Checking Parity Logic 81

Figure 49. Loopback Connectivity 82

Figure 50. Board Breaking Timing and latching (arrived at phase_12_1 ) 89

Figure 51. Board Breaking Timing and latching, (arrived at phase_l 2_2) 89

Figure 52. Board Breaking Timing on Info Loopback Error 90

Figure 53. Board Breaking Timing on Heuristic or Arbitrary Broken 90

Figure 54. Reset Signal Timing 91

Figure 55. Power Reset State Machine 93

Figure 56. Board resets 94

Figure 57. Info Bus Partitioning 96

Figure 58. board_not_broken_ routing 97

Figure 59. reset, routing 97

Figure 60. sync_ routing 98

Figure 61. online_ routing 98

Figure 62. ID PROM Implementation 99

Figure 63. Inbound Pipe Block Diagram 102

Figure 64. Post-Qual and Pre-Qual Bus Timing 109

Figure 65. data_valid and last nfo Timing 109

Figure 66. PING Control Block Diagram 110

Figure 67. Outbound Data and Control Unit 114

Figure 68. Outbound Timing Diagram 120

Figure 69. Outbound Master Control 121

Figure 70. Outbound Slave Control 122

Figure 71. Outbound Peer-to-Peer Timing 123

Figure 72. Echo Peer-to-Peer Timing 124

Figure 73. Outbound Peer-to-Peer Control 125

Figure 74. Arbitration Block Diagram 126

Figure 75. Arbitration State Machine 128

Figure 76. Arbitration Timing Diagram : 129

Figure 77. IO Register Block Diagram 130

Figure 78. Register Write Timing 135

Figure 79. Register Read Timing 135

Figure 80. Error Checking Block Diagram 136

Figure 81. Bus Error Operation Flow Chart 150

21 July 1995

_^ λ ^

Xbus Functional Specification Stratus Company Confidential

1. Introduction

This document provides a detailed description of the Polo Xbus. Contained in this document are the signal descriptions, functional protocol descriptions, functional timing diagrams, and physical bus descriptions, including pinoute and clocking methodology. The Xbus protocol is directiy based on the Stratus Gbus, and this specification is itself derived from the Gbus Functional Specification.

1.1 Applicable Documents

1.1.1 Stratus Documents

Cougar Specification Rev 12, 9/9/92, or later /auto/jetta/doc/CPU/cougar/...

Polo Programmer's Guide Rev 3.0 or later /auto/polo/doc/func_spec ...

Gofer Specification Rev 2.0 or later /auto/jetta doc/CPU/gofer/...

GottBus Specification Rev 3.1 or later /auto/jetta/doc/gbus

Mercury Functional Specification /auto/jette/doc/cpu/board/...

Polo Cyclops Specification Rev 1.0 or later /auto/polo/doc/CPU/cy clops/...

Polo Backplane Specification Rev 1.0 or later /auto/polo/doc/backplane/...

Polo Clocking Specification Rev 1.0 or later /auto/polo/doc/clock/...

1.1.2 Other Documents

IEEE Standard 1149.1 : IEEE Standard Test Access Port and Boundary Scan Architecture

February 15, 1990

Test Technology Technical Committee of the IEEE Computer Society

The Institute of Electrical and Electronics Engineers, Inc.

345 East 47th Street, NY, NY 10017-2394

21 July 1995 °

Xbus Functional us Company Confidential

1.2 Revision History

Revision 1.0:

• Changes to almost every section in order to complete document.

21 July 1995

> o

Xbus Functional Specification Stratus Company Confidential

2. Functional Overview

2.1 Introduction

The Xbus performs a number of functions:

* It supplies a means to transfer information between Polo CPU boards and PCI Bridge cards. It supplies a means to transfer information between two Polo CPU boards. It provides fault detection and isolation for the bus and the boards on the bus. However, it is only fault detecting, not fault tolerant.

This Xbus must have a number of functional characteristics. First, it must provide enough throughput so that it is not the system bottle neck. Second, it must be inexpensive to implement. Third, it must be able to support both PCI accesses, 32 bits of address and multiple words of data, and PA-RISC accesses, 48 bits of address and multiple words of data. It must also be able to support Cougar/PA-RISC functionality such as regurgitated infos, flushes, and purges.

There are a number of possible interconnect structures that could be used for this bus. The two that were carefully investigated are a fully interconnected bus, such as in Jetta or StratøBus based systems, and a series of 4 point to point buses as shown in figure 1. In order to determine which of these interconnects would best meet the Polo implementation requirements, several key areas were examined.

-f

Also, with the point to point solution, termination becomes much simpler. Instead of parallel termination series termination is used. This change in termination removes the need for generating a termination voltage and the need for backpanel power supplies.

There are some secondary considerations such as bus isolation, and complexity of implementation, but the live insertion issue is the real driver for the technology choice.

21 July 1995 10

Xbus Functional pecitication _..atus Company Confidential

Figure 1. Xbus interconnect

KEY Bidirectional, point-to-point data bus. Unidirectional, point-to-point control bus.

2.2 Architectural Alternatives

There are 4 fundamental choices for implementation of this bus. The bus could be Golfbus based, I Bus based, PCI based, or totally new. These possibilities are explored in the next sections.

2.2.1 New Bus

In order to justify inventing a new bus, there must be a good reason. If performance, functionality, or cost goals cannot be met by an existing structure, then it is necessary to invent one. This is not the case for Polo. Therefore, this possibility is discarded.

2.2.2 PCI Based Bus

A PCI based bus is one possible implementation alternative. It meets performance and cost goals. It could be modified to meet the addressing and Cougar functionality. The PCI is probably not quite optimal from a performance stand point due to the tact that it is not split transaction. This is clearly a negative point for a PCI based solution. This design probably simplifies the Gambit ASIC at the

21 July 1995 11

Z o l

Xbus Functional Specification Stratus ompany Confidential

expense of the Cyclops ASIC.

If a PCI based bus is used, the PCI protocol will have to be modified to support the virtual index of the PA chip. This is required to support CPU to CPU transfers. The PCI protocol will also have to be modified to support the fault detecting nature of the interconnect. Finally, some new function codes will have to be added to the bus in order to allow it to support the sync functionality of the CPU boards.

For these reasons, a PCI based solution is discarded.

2.2.3 Ibus based Bus

The Ibus based implementation has quite a few advantages over the previous two possibilities. The Ibus is already defined and has a performance and cost point that meet the needs for the Xbus. it already supports PA virtual indexes, and a split transaction protocol. It will require modification to support PCI based non virtual index transfers, but there are spare bits in the virtual index transfer phase that can be used for this purpose. So inventing that cycle should not be difficult.

There are some disadvantages to an (bus based solution. First, an error protocol will have to be invented that can support the fault detecting needs of this bus. This will require reworking some of the central state machines of the Ibus. Second, although the Ibus has some similarities to the Golfbus, there are enough differences that it is unlikely that the RTL recycling inside of Gofer will be large. The tie between the Ibus and the Golfbus inside of Gofer is tight, therefore, removing the Golfbus may require a substantial reworking of the Gofer internal interfaces.

For these reasons, although the Ibus based solution is a reasonable one, it is not considered to be as good as the next possibility.

2.2.4 Golfbus Based Bus

Although at first glance a Golfbus based solution for the Xbus may seem like overkill, it has a number of significant advantages even over the Ibus based solution. The Golfbus maintains all of the Ibus advantages. However, the Golfbus also has the advantage that it already includes a robust enor handling protocol. It has an interface that is well integrated into the existing Gofer RTL logic. Hopefully, a Golfbus based solution will allow for a maximal sharing of RTL code between Gofer and Cyclops. Additionally, the Golfbus logic has already been used in a common logic fashion. This may allow for good sharing of logic between the Cyclops ASIC and the Gambit ASIC.

Xbus will use a Golfbus-based protocol running on a newly designed, point to point

2.3 Golfbus Logical Overview

The Golfbus is a single logical split transaction multiplexed address/data bus. It is duplicated to provide fault tolerance in a manner similar to the Stratabus. Major features of the bus include the ability to support 32 and 64 bit bus interface widths, completely synchronous operation, cache consistency support, and a single logical ASIC interface.

In its initial implementation, the bus supports -

• 2 logical (4 physical) CPU/memory boards, with a 64 bit bus.

21 July 1995 ^ ~ 12

Xbus Functional ΛJS Company Confidential

• 10 physical IO boards, with a "32 bit" bus.

Peak supported bandwidths are 128MBytes/sec between CPU/Mem boards (assuming all line transfers) and 76.8MBytes sec to/from IO boards. All information transfer across the Golfbus occurs synchronously to/from the bus transceivers at ~24MHz.

A single logical bus is used for both address and data transfer. The bus contains up to 64 bits of information (data or address and function code), 7 bits of tag, and a single bit which indicates whether the information lines are canying data or address and function code. The IO boards in the initial implementation will only connect to 32 bits of the information bus. (Special function codes will be used to interface to and from the IO boards.) The information bus, tag, and func.op bit are covered by 8 or 16 bits of parity for "32 bit" interfaces and "64 bit" interfaces respectively.

Control signals and arbitration lines are protected by a three-way voting algorithm. This provides the ability to tolerate a single control signal failure within each three-way voted control line.

The Golfbus interface logic provides full checking between the C-side and D-side of each Golfbus board via loopback, thereby providing board level fault detection. The Golfbus interface is able to isolate itself from the Golfbus, even in the event of clock failures, thereby providing board level fault isolation.

Because of the split nature of the bus, all boards in the system must be able to arbitrate for the bus and initiate a bus operation.

2.4 Golfbus Physical Summary

The Golfbus has an extremely efficient physical implementation. Support of its information transfer and fault detection isolation capabilities require (in addition to clock generation circuitry) only:

• Connectors

• Transceivers

• 1 logical (2 physical) ASICs

1 logical (2 physical) MSI 20-pin register component

Support of board ID PROMs and board indicator circuitry (card edge LEDs) requires some additional MSI circuitry.

The 32-bit version of the bus interface requires fewer than 150 backplane signal connections and the 64-bit version of the bus interface requires fewer than 250 backplane signal connections.64- bit and 32-bit interfaces coexist in the same backplane, though backplane slots are wired to support one or the other. The AMP SL-100 connector is used for the Golfbus backplane connections.

The transceiver used on the Golfbus is the FB2033, a Stratus specified bicmos device provided by Signetjcs Corp. and Texas Instruments Inc. This is a Futurebus compatible device with controlled edge rates, capable of delivering 100mA on the bus side. These octal devices are implemented as a three port device, i.e. each bit has a board side input port, a board side output port, and a bus side bidirectional port. The transceiver is used in a clocked mode for both inbound and outbound data flow. A testability feature, which allows internal loopback checking, i.e. a path that includes the entire part except the bus side output driver and input buffer, is also utilized to facilitate testing. The transceiver supports live insertion and removal.

21 July 1995 13

Oo

Xbus Functional Specification Stratus company Confidential

"64-bit" Golfbus interfaces can be supported via 2391 -pin ASICs and "32-bit" Golfbus interfaces can be supported via 2 304-pin ASICs, though for the first implementation the former will be used for both.

The Golfbus interface requires three clocks to transfer information across the backplane, a -24MHz transmit clock, a -24MHz receive clock, and a -12MHz phase information clock qualifier. In addition, first generation Xbus interfaces will require 2 ~24MHz clocks (really OE signals) for transferring information from the bus transceivers to the bus ASICs.

2.5 Polo XBus Logical Overview

Polo implements a subset of the of the Golfbus protocol. The following discussion outlines key Golfbus features not supported in the Polo implementation.

Bus widths are limited to 32 bits. Since there are no CPU boards with remote memory on the Polo Xbus, there is no need to use the optional 64 bit widths of the Golfbus.

The configurations are limited to 4 devices. Two CPU boards and two PCI bridge cards.

Although the physical implementation uses 4 point to point buses instead of 2 fully connected buses, the protocol is still implemented with a single bus view. On any given cycle, there is no more than one transaction on the interconnect. This transaction may be on all 4 buses for the case of a duplexed operation or on only 2 of the buses for a simplexed operation. It is the responsibility of the arbitration network to ensure this functionality.

Like the Jetta implementation, the signals are divided into two categories, transaction based bused signals and control signals. However, due to the point to point nature of the signals, the implementation is very different. The bused signals are implemented as 4 bidirectional point to point buses. These are shown as the heavy lines in figure 1. The control signals are implemented as separate unidirectional buses. These are shown as the lighter lines in figure 1. The bused signals are protected through the map ram, parity, and loopback checking. The control signals protected through ECC and loopback checking.

One fundamentally new feature of the Xbus is the error protocol. Unlike the Golfbus, the Xbus attempts not only to detect bus errors, but also to diagnose the source. This aspect of the Xbus is covered in a later section in detail.

A second fundamentally different feature is the mechanism by which two CPUs communicate. A CPU to CPU transaction is accomplished by means of a peer-to-peer operation in which the transaction is broken into two separate transactions. In the first, the CPU transmits the transaction from the CPU to the PCIB. In the second, the PCIB transmits the information from the PCIB to the CPU. This is explained in detail in the section on peer-to-peer transactions.

2.6 Polo XBus Physical Overview

The Polo XBus physical implementation makes use of a direct ASIC to ASIC connection technology. The bus interface is accomplished through the use of 2391 pin PGA package ASICs. The Xbus uses two different styles of connectors. The CPU boards (suitcases) use the SL-100 connector from AMP. This connector was chosen for its ruggedness, built in stiff eners, and grounding scheme. The PCIBs use the multiple sourced tuturebus connector. These connectors, while less rigid, provide good signal integrity, and are inexpensive. A single 24Mhz clock is used for clocking data and control onto and off of the bus.

21 July 1995 14 '

Xbus Functiona. - peciiιoation .atus Company Confidential

3. Detailed Functional Description

3.1 Bus Naming Convention

The Xbus actually consists of 4 data buses and 12 control buses. With so many buses in the system, it is important to have a concise intuitive naming convention.

3.1.1 Data Bus Naming Convention

Figure 2 below, shows a block diagram of the 4 data buses in the Xbus system. Figure 2. Xbus data Buses

The number for the bus is taken from the CPU slot number; therefore date buses connected to CPU 0 end in 0 and data buses connected to CPU 1 end in 1. The letter for the bus is determined by whether or not the bus is a crisscross bus (i.e. connects an even slot number to an odd slot number) or a straight bus (i.e. connects an even to an even or an odd to an odd slot). Based on this convention, CPU 0 has connections to data bus AO and BO. CPU 1 has connections to data bus B1 and A1. PCI Bridge 0 has connections to data bus AO and B1. PCI Bridge 1 has connections to BO and A1. Note that unlike traditional Stratus boards, the PCI bridge cards do not run in lock-step.

21 July 1995 15

^ O

Xbus Functional Specification Stratus C-.npany Confidential

3.1.2 Control Bus Naming Convention

Figure 3 below shows a block diagram of the control buses in the Polo system. Figure 3. Xbus Control Buses

The control naming convention for the backplane signals uses the signal name followed by the source of the signal and then the destination of the signal. The naming convention for the associated ASIC pins uses the signal name, the direction (in or out), and the connection (n for neighbor, o tor opposite, and p for peer). Examples of this naming convention are shown in Figure 3. Table 1 lists all of the control bus names.

Table 1. Control Bus Names

21 July 1995 16

T Λ

Xbus Functiona. -pecitication atus Company Confidential

3.2 Bit Numbering

The Xbus and HP-PA7100 use different bit numbering conventions. Refer to Figure 4 for a description of the bit numbering scheme used on the Xbus. Refer to the Mercury Functional Specification for a description of the HP-PA7100 bit numbering scheme. Refer to the MIO/Polo PCI programming specification for the PCI bit numbering scheme.

Figure 4. Xbus Bit Numbering Convention

increasing order ol significance

Xbus data bits 31 2423 16 15 87 0

Big-Endian byte O byte 1 j byte 2 byte 3 (Xbus)

3.3 Terminology bus cycle — the 24MHz (-41.67 ns) building block from which alt Xbus operations are built A bus cycle is the time which a valid logic level driven by one board on the backplane is seen by all other boards. Two bus cycles compose a bus phase, 4 bus phases compose a bus operation, and one or more bus operations compose a bus transaction. bus phase — the 12MHz (-83.3ns, 2 bus cycle) building block from which all bus operations are constructed. There are logically 11 types of bus phases on the Xbus; "Art)", "Info", "Postl", and "Post2" are the phases that occur during normal operation. When an error is detected, the special phases "Errorl", "CPU test", "CPU Post ^* , IO Test', IO Postl ", "IO Post2", and "Error?" are inserted in the protocol for fault isolation. (The error phases are sometimes collectively referred to as "Post3"). During each bus phase it is possible to transmit two sets of information on a physical set of backplane lines (i.e. "double pumping") though this is not done for all bus phases and/or signals. bus operation — A bus operation is the basic unit of address and data transmission and checking on the Xbus. It is generally composed of at least 4 phases: an Arb phase followed by Info, Postl

21 July 1995 17

%o

Xbus Functional Specification Stratus company Confidential

and Post2 phases. Bus errors cause the insertion of the error phases after Post2 and increase the number of phases required to complete a bus operation. Bus operations may consists of multiple info transmissions in the case of a block transfer. A bus operation can be thought of as a full one way transfer on the Xbus. bus sub-operation — A bus sub-operation is an operation initiated by a bus master during subsequent phases of a block transfer. Sub-operations always cany data and BUSYs are ignored. It is generally composed of 4 phases: an Art) phase during which grant inhibit is used, followed by Info, Postl and Post2 phases. Bus errors may increase the number of phases required to complete a bus sub-operation. A sub-operation is differentiated from an operation in that a bus operation for a block transfer consists of the first transfer plus a number of sub-operations consisting of multiple data transfers. bus transaction — a complete high level exchange of information on the Xbus. Examples include reads and writes. A read transaction between CPU and PCIB is composed of a minimum of two bus operations; one operation provides the address and function code, and one or more operations provide the retum data. A write transaction to a PCIB is composed of a minimum of two operations; one operation provides address and function code, and one or more additional operations provide the data.

The following diagrams illustrate the terminology surrounding a simple word read and line write transaction.

Figure 5. Terminology - Simple Word Read Transaction

I OODϋOOOO,

A four phase bus operation. In the word A four phase bus operation. In the word read transaction shown here this first read transaction shown here this first operation is used to transmit tt e read operation issued to transmit the data. address and function code.

A complete bus transaction. This example simple word read transaction is composed of two operations, one to transmit address and one to transmit data.

21 July 1995 18

Xbus Functional opecitication _^ ..atus Company Confidential

Figure 6. Terminology - Simple Line Write Transaction

A 12MHZ (-83.3ns) bus phase (composed of 2 bus cycles) v _y >

A four-phase bus operation v _y ,

A four-phase bus sub-operation _v >

A complete bus transaction. The simple word write transaction shown is composed of two over-lapping operations.

Bus Master — A board that has won arbitration. This board drives the info lines in the info phase of the bus operation. A bus master can be a transaction master or a transaction slave.

Bus Slave — A board that has determined the info lines carry information that it must receive. A bus slave can be a transaction master or a transaction slave.

CD different read — A read in which the C and D side ASICs each provide half the data, e.g. when reading error status registers. Loopback checking of the bytes driven by the other side is suppressed.

Cyclops - The Xbus to Ibus interface ASIC on the CPU board in Polo systems. echo transaction — The second half of a peer-to-peer bus transaction between CPUs. Send and echo transactions are not split; grant inhibit is used to ensure that no other transactions occur between the send and echo. This is to prevent re-ordering.

EFQ - Eviction-Flush Queue. This queue exists only on CPU boards. Refer to the Cyclops (Bus Interface) Specification for details.

EFQ-Freeze-State - Set via bit 19 of the Bus Interface State Register. This mode only exists on CPU boards. When in this mode, a board will busy all accesses directed to its EFQ except those from its partner unless it is a data retum of any size.

Fast Duplexing - Fast duplexing is a form of duplexing in which no memory is updated. This is done when both boards are coming up for the first time as opposed to updating a new board from a running OS.

Freeze-State- Set via bit 14 of the Bus Interface Stete Register. This mode only exists on boards that can be duplexed. When in this mode, a board will busy all accesses directed to its RWQ

21 July 1995 19 l o

Xbus Functional Specification Stratus company Confidential

except those from its partner.

Gambit - The Xbus to PCI ASIC on the PCI Bridge card. IT interface to the Xbus and the PCI bus.

I/O virtual address (IOVA) - An IOVA is an address generated by a PCI card. This address is transmitted across the Xbus to the cyclops ASIC. Inside the cyclops ASIC the address is translated into a valid system address. The IOVA is used to provide fault tolerance. It guarantees that a PCI card will generate a correct address range. loopback checking— This is when an ASIC checks that the value it sees on a pin is equal to the value it thinks should be on the pin during normal operation. loopcheck operation— This is when the bus ASICs drive 55/AAs as part of the error protocol in order to determine the site of a fault. peer to peer transaction — A two part transaction between two CPUs. The Xbus does not have fully interconnected data buses, and transfers between the two CPU boards must occur in two steps: first a send between the CPU and the PCIB(s), then an echo from the PCIB(s) to all of the CPU(s). The requesting CPU drives a complete transaction on its A and B buses. The PCIBs look at the address, and determines whether the transaction is directed to the CPUs. If it is, they buffer the transaction in order to repeat the transaction on their A & B buses once Post2 of the last info phase has passed with no bus errors. In this way, both CPUs see the transaction at the same time. Peer to peer transactions between CPU boards require a minimum of four operations.

RWQ - ReadWrite Queue. This queue exists only on CPU boards. Refer to the Cyclops (Bus Interface) Specification for details.

Regurgitated Info - A regurgitated info is a cougar generated cycle used during the update process. It is generated by the update-on-line board and transmitted to the update off-line board. It is unique because an update off-line board accepts info cycles even if the base address does not match the base address of the board. send transaction — The first half of a peer-to-peer bus transaction between CPUs. single side operation — An operation with data supplied entirely by either the C or D ASIC; e.g. a read from a PCI card. Loopback checking is performed only by the side supplying the data.

TRID — Transaction identifier. This a unique binary number used to tie together two bus operations during a split transaction and to identify the source for write transactions (TRIDs on write transactions are strictly for debug). A TRID is unique only while a given transaction is still outstanding and will be re-used for later transactions by a transaction-master. Note that trid bits 02 - 00 are used for the slot number of the transaction-master and trid bits 06 - 04 are generated by an on-board master - thus allowing a board to have 8 unique masters with transactions outstending.Trid bit 03 is a new bit for Polo that indicates the format of the address; a zero indicates a Jetta style system address is being transmitted and a one indicates an IOVA (I/O Virtual Address) format is on the backplane. The IOVA address needs to be translated into a system address via the map RAM. Refer to section 6.10.1 for a complete description of the trid field.

Transaction Master — The specific resource on a board that generated a transaction. The transaction master is responsible for generating the TRID of the transaction.

Transaction Slave — The board on which a transaction was directed towards, i.e. a board in which the function code and address of a bus operation has decoded to.

21 July 1995 , ~ 20 i )

Xbus Functional .pecιtιυation . atus Company Confidential

4. Xbus Signal Description

The following section describes in detail the vanous signals that comprise the Xbus. As there are 4 buses in the Polo implementation, a new naming convention has been developed for the buses. This naming convention ts described in section 3.1, Bus Naming Convention, on page 15.

A functional description is included for each type of signal and not for every individual signal (for example: there is a TRID field for ail 4 buses, however there is only one description that covers both). Power control and JTAG signals are not described here.The signal names are designed to be compatible with the Verilog coding standards so that the specification will match the code exactly. The following rules were used in creating the signal names: all lower-case characters the "_" is a delimiter when it is not at the end of a signal name the "_" at the end of the signal name indicates that the signal is low-true

_a indicates A_bus signals

_b indicates B_bus signals

_x_ _y_, and _z_ are used for the three low-true signals on a triplicated net

[n:m] is used to describe a multi-bit field

4.1 Signal Description

As described earlier, the Polo Xbus implements the control and bused signals very differently.

The bused signals are implemented as 4 point-to-point bidirectional signals. The ability to drive these signals is controlled through the arbitration network described later. These signals are protected by a single parity bit that accompanies the bus. The buses are interconnected so that each CPU is connected to both PCI bridge boards and each PCI bridge board is connected to both CPUs. Error recovery is accomplished through the XBus error protocol.

The control signals are organized in a set of point to point unidirectional buses. Each of these buses is ECC protected. These buses cany control signals which are not governed through arbitration. Unlike the bused signals, there is a control bus in each direction between every board. This is necessary in order to ensure the single bus view of the system. For example, if one PCI bndge card sees a bus error, that information must be transmitted to all three ottier boards in order for the boards to all perform the error sequence.

The bused signals and control signals are double pumped at 24MHz each cycle. That is, they carry different data dunng the first and second 24Mhz bus cycles that make up a single phase. All buses and control signals are active high on the backplane.

A small number of reset and broken related control signals are buffered by the 26S10 transceiver and replicated for three way voting. These signals are active low on the backplane.

4.2 The Info Bus

The following signals are collectively referred to as the info bus. Although there are actually 4 sets of these signals (a0,a1,b0,b1), for simplicity's sake only the aO version is listed. For example, when the TRID field is described, it should be understood that there are actually 4 TRID buses:

21 July 1995 21 l 't

Xbus Functional Specification Stratus Company Confidential

trid_a0, trid_b0, trid_ai , and toϊd_bl . Table 2. Xbus Bidirectional Buses

4.3 The Control Bus

This section describes the control signals. There are actually 12 control buses, but again only one is described here. The names of the twelve control buses are listed in table 1. For simplicity of documentation, the control bus is identified by bit numbering, similar to the trid field. However, since the meaning of the control bus bits is very significant, each one is described in deteil here.

21 July 1995 22

2Λ

Xbus Function.., Specmcation tratus Company Confidential

The control buses are protected by a single bit correction, double bit detection ECC code. Table 3. Control Bus Signals signal width description control[01 12 bus_req and ack - During the first half of the phase, this bit is used for (12x1) bus_req. During the second half of the phase, this bit is used for ack.

During the first half of the phase, this bit is driven by a board when it is requesting the bus. As described in later sections, the bus uses a distributed arbitration model loosely based on the Golfbus. Each board in the system drives the bus.req and tests all of other boards bus_req signals to determine who will drive the info signals during the next phase.

During the second cycle of the phase, this bit is used to acknowledge a bus transaction. Ack is asserted in Post2 by the target board of the transaction. This signal is the result of an address decode, so it is only valid in Post2 of an operation that is transferring an address. Ack provides an indication of whether or not a transaction is progressing. This is relevant in a Polo system, since a PCI card may go away resulting in a no ack for a ping operation.

Acks in a Polo system also let a PCI Bridge know that a CPU's map RAM has mapped a PCI initiated access to a valid CPU address. If a PCIB's read or write is not ACKed, the PCI slot that initiated the access may or may not be set off-line depending on bits in the Gambit's configuration register, described in section 12.6.4.9, SAM Configuration, on page 127 in the Polo Programming Guide.

Writes from the CPU to the PCIB are acked to facilitate debug, but are otherwise unused. A peer to peer CPU write is not ACKed by the PCIBs, but the echoed operation is acked by both CPUs; again, this is only for assisting debug.

Ack is ignored for the send portion of a peer-to-peer operation. The CPU initiator of a peer-to-peer operation must track the entire operation to see whether the cycle is acknowledged.

21 July 1995 23 l^

Xbus Functional Specification Stratus company Confidential

Table 3. Control Bus Signals

21 July 1995 24 a

Xbus Functional _.peciτtcation atus Company Confidential

Table 3. Control Bus Signals

4.4 The Voted Signals

The voted signals are the only signals driven through 26S10 transceivers. These signals are point to point and terminated at each end, so that insertion of an unpowered board does not disturb the

21 July 1995 25

2

Xbus Functional Specification Stratus company Confidential

termination (and timing) of a net in use. Only the _x_ versions are listed; there are _y_ and _z_ Table 4.3-Way Voted Signals

o k

J v

LP

K

signals making up the triplet

21 July 1995 26

Xbus atus Company Confidential

4.5 Other Control Signals

Table 5. Miscellaneous Control Signals

21 July 1995 27

S

Xbus Functional Spec .if..ica _ti. _Λ o„n ._ S_tr_atus _c omp _aLn_y — Co _nfidential

5. Xbus I/O Interface Registers

A complete description of the Polo register set can be found in the Polo Programming Guide.

21 July 1995 28 5

Xbus Functional pecincation _^ ..atus Company Confidential

6. Xbus Protocol

6.1 Overview

The Xbus is a point-to-point synchronous, pipelined, multiplexed address/data, error detecting bus with split-transaction capabilities. Function-code, address and data are parity and loop-back checked. Control lines are ECC protected and loop-back checked. Three-way voting is implemented on the reset, clock, and broken indicator lines.

The bus supports word accesses and has a block transfer capability for support of cache line accesses. The Xbus has a logical 32-bit wide data/address bus.

6.2 Bus Operation

The basic component of all Xbus transactions is the operation. An operation is composed of four phases as illustrated in figure 7: arb, info, postl , and post2. Two information transfers can occur on the bus during each phase; this is referred to as "double pumping". The double pump frequency is approximately 24MHz. The figure below illustrates the logical activity on the bus. All information is directiy registered in the ASICs without any external buffers.

Figure 7. Basic Xbus Cycle

83.3ns

" virtual index, function code, remote/coherent bits, and byte enables or I/O

The phases are used as follows:

Arb phase: boards drive their arbitration request lines during the first half (cycle) of the arbitration phase. During the second half they determine whether they won arbitration and prepare for the info phase.

Info phase: For non-IOVA address transfers, boards drive the virtual index, function code, remote coherent bits, and byte enables during the first half of the info phase and the physical address during the second half. For IOVA address transfers (IOVA bit in the trid is true), boards drive the IOVA during the first half of the info phase and deterministic data with good parity during the second half; the physical address is gotten from the I/O address map RAM look-up. For data transfers, data is driven during both the first and second halves of the cycle. Note that non-cache consistent address transfers need not supply a virtual index though the driven information must be deterministic and parity will be computed across it.

21 July 1995 29

2Λ

Xbus Functional Specification Stratus company Confidential

Postl phase: During this phase, boards are determining whether any error conditions existed with the info phase and whether there is need to BUSY the operation. CPU boards map the device index portion of the IOVA to obtain the full physical address and virtual index of an I/O board's transfer for IOVA address transfers.

Post2 phase: Any board detecting an error with the info phase drives the error lines during the first half. If a board does get errored, it next goes to the error sequence phases to determine the source of the error. Any board detecting the need to BUSY an address/function code driven during the info phase drives BUSY during the second cycle of this phase. It is also during this phase that accesses are acknowledged (this is described in greater detail in someothersection) .

6.3 Bus Busies

Figure 8 illustrates the effect of bus BUSYs on the basic bus cycle. As shown in the figure, BUSY has no effect on a bus operation during any phase except for post2; a BUSY during post2 will cancel the bus operation. Note that busys for multiple cycle bus operations, such as block transfers, have special rules and are described in section 6.7.4 on page 49.

Should a cycle be both BUSYed and ERRORed, the ERROR takes precedence. Figure 8. Basic Bus Busy Operation

6.4 Bus Error ^• ss PrWrCCrl

This section covers bus errors on the info bus, for information on the control bus see Section 9.2 on page 84.

Figure 9 shows the effect of bus errors on the basic bus cycle.

21 July 1995 30

Xbus Functiona. deification itus Company Confidential

Figure 9. Basic Bus Error Operation

following execution of the error protocol. The arbitration is ignored in the previous cycle.

Since the Xbus has no transceivers, the loopcheck phase of the Golfbus error protocol (Post4) has been modified to allow each board an opportunity to verify its transmit and receive capabilities. This has resulted in new states being added to the bus error operation. These states are described below:

Errl: The Errl stete is entered on the cycle after a bus error is detected. This state is used to allow for time to turn off the info bus before the loopback checks are performed. A board that is in its info phase during Errl will disable its output enables half way through the phase.

CPUTest: The CPUTest state is used to test the CPU's ability to drive patterns on the Xbus. On the first cycle of the phase the CPU will drive 55 on the info bus, 55 on the trid bus, 1 on the parity line and 0 on the func_op line. On the second cycle of the phase the CPU will drive AA on the into bus, 2A on the trid bus, 0 on the parity line and 1 on the funcjop line.

CPUPost: The CPUPost state is used to turn the bus around between the CPU's loopback check and the I O boards loopback check. This phase is also used as a Postl cycle for the CPU's loopback pattem.

IOTest: The IOTest state is used to test the I/O board's ability to drive patterns on the Xbus. On the first cycle of the phase the I/O board will drive 55 on the info bus, 55 on the trid bus, 1 on the parity line and 0 on the func.op line. On the second cycle of the phase the I/O board will drive AA on the info bus, 2A on the trid bus, 0 on the parity line and 1 on the funcjop line. Bus errors from the CPUTest phase are reported during this phase. This information is used to evaluate the bus, CPU, and I/O board at the end of the error sequence. The last I/O ASIC to drive the data bus drives the bus during the IOTest phase. lOPost: The lOPosti state is used to evaluate the IOTest data.

21 July 1995 31

1 ^'L

Xbus Functional Specification Stratus -.npany Confidential

IOPost2: The IOPost2 state is used to transmit any bus errors from the lOPostl stete. This information will be used to make an intelligent decision about how to deal with the error.

Err2: The Err2 state is used to evaluate the infomiation from the loopback checks. Bus errors from CPUPost and IOPost2 as well as information shared between the C and D sides of each board are used to determine what course of action to take. This set of actions will be described later in this section.

Figure 10 shows the basic stete machine and state transitions for the bus error handler.

The key challenge for the bus error algorithm on the Xbus is to diagnose errors so that system operation can continue. Unlike previous systems that use duplicated buses to allow all functional units a guaranteed path for communications, when the Xbus removes a bus from service, it must also remove one of the twpMβit$attached to that bus, in some cases, the right thing to do is obvious. In other case^ηis notJrhe following sections analyze various faults, how they are handled and now tπβy manifest themselves.

6.4.1 Bus Error Broken Conditions

At this point it would be helpful to classify the different types of conditions that cause a board to go broken when a bus is detected bad. For a full discussion of the broken logic, see Section 9. on page 80.

21 July 1995 32

I

Xbus Functions ^pecmcation atus Company Confidential

• loopback on control - C and D ASICs must always agree on what to drive on the control lines, including whether or not to assert bus error. If one ASIC asserts bus error and the other side does not, the board breaks.

• loopback on data - C and D ASICs must always agree on what to drive on the duplicated info lines. When driving CD same data, ASICs compare the date they drive with the date they receive. An ASIC asserts bus error on parity errors when receiving data, and parity errors and loopback errors when driving data. Loopback checking is disabled when a board drives "CD different" data, such as the contents of error reporting registers or date from PCI cards. The board breaks if the two sides disagree on which bytes have or do not have errors,.

• arbitrary - Break the designated board in Err2 when there are bus errors signaled during CPUtest and iOtest and no board has broken by the end of IOpost2. This is called an arbitrary shoot because the fault is most likely on the backplane, so it is arbitrary as to which of the two boards connected to the faulty bus to break. Typically, the CPU is set broken, so that the system can continue with all of its I/O available, but if bit 21 of the Bus Interface State register (Section 12.3.4 on page 87 in the Polo Programming Guide) is set then the PCIB board will be the designated board.

• heuristic - a board breaks itself during Err2 if there is a bus error when it drives, but no bus error when the other board drives, and the other board did not break by the end of IOPOST2.

6.4.2 Xbus Fault Analysis

In order to understand various faults and what they can mean, it is important to present a deteiled block diagram of the Xbus interconnect Figure 11 show the interconnect for a typical Xbus line.

The black dots in figure 11 represent the connectors to the backplane. For fault tolerance and fault isolation reasons, it is important that the boards should be routed so that the etch between the D-

21 July 1995 33

°

Xbus Functional Specification Stratus _ jmpany Confidential

side and the C-side runs through the connector connection. This limits the amount ot etch on each board that cannot be isolated øttfe to a minimum. On the CPU board, one ASIC both drives and receives a given net while the other ASIC only receives that net On the I/O board, each ASIC can potentially drive every net The CPU ASICs are always in lockstep and therefore each ASIC is capable of sharing the data out load. However on the I/O board, each ASIC connects to a different PCI bus so a signal ASIC may need to drive the entire Xbus. There are cases in normal operation when only one CPU ASIC will drive the entire bus.

6.4.3 Fault Conditions

The following sections identify all known fault conditions and describe their handling. Refer to figure 11 to determine the location of the fault site indicated.

6.4.3.1 CPU Board Faulty Input Circuit - CPU Driving

• fault site 1

• break via loopback on control fault

This fault deals with a fault in the input section of one of the CPU ASICs. In this case, the fault occurred during or just before a cycle in which the CPU drove the info bus. The error is detected when the CPU drives the bus. The ASIC with the faulty circuit will signal a bus error during the Post2 phase of the cycle and the other side ASIC will not. The board will go broken and drive bus error during Errl . The error sequence will be executed, and the operation will be retried by the partner CPU with no error.

6.4.3.2 CPU Board Faulty Input Circuit - I/O Board Driving

• fault site 1 break via loopback on control fault

This fault deals with a fault in the input section of one of the CPU ASICs. In this case the fault occurred during or just before a cycle in which the I/O board drove the info bus. If the error is a multi-bit error that evades the parity logic, the error will be caught internal to the CPU board and the CPU board will go broken. If the error is a single bit error the faulty ASIC will detect a bus error during the Postl phase of the transfer. The ASIC will drive bus error during Post2 of the transfer and the otherside of the board will not. The board will break with a loopback on control failure in tiie next phase. After the error sequence, the operation will be retried by the partner CPU with no error.

6.4.3.3 CPU Board Different Data C-Side and D-side

• fault site 2 break via loopback on data fault

This fault deals with an internal CPU fault that results in different date being driven out of each ASIC. The error is detected when the CPU drives the bus. The C and D ASICs will trade error status and disagree on where the error is during Postl; both C and D sides will see an error on bytes the other side drives but no error on the bytes it drives. The board will go broken and drive bus error during Post2. The error sequence will be executed and the operation will be retried by the partner CPU with no error.

6.4.3.4 CPU Board Faulty Output Circuit - Buffer to Pad Fault

• fault site 3

21 July 1995 ~~ 34

Xbus Functional specification atus Company Confidential

break via heuristic broken

This is a fault in the output section of the CPU ASIC resulting from an output driver circuit fault that blows in a manner that causes an internal open between the output driver and the ASIC pad, while not disrupting the functionality of the input receiver. All ASICs on the bus will detect a bus error during the Postl phase of the transfer. The ASICs will drive bus error during Post2 of the transfer. All ASICs will detect a bus error during the CPUPost phase. No bus errors are detected during the IOpostl phase. The CPU board will go broken during Erτ2 based on the fact that it detected a bus enor when it drove the bus, no bus error when the I O board drove the bus and the I/O board did not go broken after its error sequence.

6.4.3.5 CPU Board Open • CPU Board Driving

• fault site 4

• break via loopback on data fault

This fault results from an open due to either a broken etch or a lifted pin on the CPU board.The routing is very important for this class of fault The etch between the C-side and the D-side should be routed through the connector pin. This limits the possibility that an open on the CPU board is mistaken to be an open on the backplane. In this case the fault occurred during or just before a cycle in which the CPU drove the info bus. The error is detected when the CPU drives the bus. During the Postl phase, the driving ASIC will not see an error but the checking ASIC will signal a compare error. During Post2 the board will go broken and drive bus error. The operation will be retried by the partner CPU with no error after the error sequence.

6.4.3.6 CPU Board Open - I/O Board Driving

• fault site 4 break via loopback on control

This fault results from an open due to either a broken etch or a lifted pin on the CPU board. The routing is very important tor this class of fault The etch between the C-side and the D-side should be routed through the connector pin. This limits the possibility that an open on the CPU board is mistaken to be an open on the backplane. In this case the fault occurred during or just before a cycle in which the I/O board drove the info bus. The ASIC with the open between it and the connector will detect a bus error during Postl . During Post2 one ASIC will assert bus error and the other will not, causing the board to break. The operation will be retried by the partner CPU without any error after the error sequence.

6.4.3.7 CPU Board Short

• fault site 4

• break via arbitrary broken

This fault deals with a short on the CPU board. When the fault occurred and who was driving the info bus during the fault are not relevant to this class of fault During Postl of the transfer all ASICs on the bus will detect a bus error. The ASICs will drive bus error during Post2 of the transfer. Both ASICs on the CPU will detect a loopback error during the CPUPost phase and the ASICs on the 1/ O board will signal a bus error during IOTest The ASICs will detect a bus error during the IOpostl phase and signal a bus error during IOPost2. During Err2 the designated board will go broken based on the fact that it has detected bus errors during the error sequence and no other boards went broken after the error sequence.

21 July 1995 ?2 6 35

Xbus Functional Specification Stratus company Confidential

6.4.3.8 Backplane Open Etch

• fault site 5

• break via arbitrary broken

When the fault occurred and who was driving the info bus during the fault are not relevant issues for this class of fault During Postl of the transfer some ASICs on the bus will detect a bus error and drive bus error during Post2. Both ASICs on the I/O board will detect a bus error during CPUPost The CPU ASICs will detect a bus error during IOpostl . During Eπ2, the designated board will go broken based on the fact that it has detected bus errors during the enor sequence and no other board went broken during the error sequence.

6.4.3.9 Backplane Short

• fault site 5

• break via arbitrary broken

When the fault occurred and who was driving the info bus during the fault are not relevant issues for this class of fault All ASICs on the bus will detect a bus eπor during Postl and drive bus error during Post2. All ASICs on the bus will detect a bus error during CPUPost and IOpostl . During Err2, the designated board will go broken based on the fact that it has detected bus errors during the error sequence and no other board went broken during the error sequence.

6.4.3.10 I/O Board Faulty Input Circuit

• fault site 6

• break via loopback on control

This fault deals with a fault in the input section of one of the I/O Board ASICs. For this particular fault, it is irrelevant who was driving the backplane when the fault was detected. The faulty ASIC will detect a bus error during Postl . During Post2 of the transfer, the faulty ASIC will drive bus error and the other ASIC will not, causing the board to go broken. If it was a CPU initiated request the operation will be retried by the CPU with no error after the error sequence. If it was a request initiated by the I/O board, then the request will be dropped.

6.4.3.11 I/O Board Output Circuit Fault - Buffer to Pad

• fault site 7

• break via heuristic broken

This is a fault in the output section of the I/O board ASIC. This class of fault results from an output driver circuit fault that blows in a manner that causes an internal open between the output driver and the ASIC pad, while not disrupting the functionality of the input receiver. All ASICs on the bus will detect a bus error during Postl of the transfer and drive bus error during Post2. No error is detected during the CPUPost phase. The I/O ASICs wili detect a bus error during IOpostl . During Err2 the I O board will go broken based on the fact that it has detected a bus error when it drives the bus, no bus error when the CPU board drove the bus and the CPU board did not go broken after its error sequence.

6.4.3.12 I/O Single-side Access - ASIC Parity Gen. Fault

• fault site 8 break via internal parity generator broken

A fault in the parity generation logic during single side accesses (data driven entirely by either the

21 July 1995 ? "") " ³⁶

Xbus Functional ~,jecifιcation _^ jtus Company Confidential

C or D ASIC) could cause the system to bus error forever. The otherside ASIC doesn't know the data, and has no way of checking the parity. For this reason, the info bus parity generation is duplicated and selfchecking inside of the Gambit ASICs. See section 9.1.1 , Parity, on page 80 for details.

6.4.3.13 I O Single-side Access - ASIC PCI Data Path Fault

• fault site 8

• break each PCI slot due to checksum and map RAM errors.

This is a fault within the ASIC's PCI data path. This hardware is not running in lockstep with the other side of the board and therefore is not self-checking. Eventually bad addresses produced by the PCIB will cause map RAM errors and/or data checksum errors in the CPU ASIC. The CPU ASIC noACKs accesses that cause map RAM errors, and the PCIB sets off-line (breaks?) the PCI slot that originated the noACKed cycle based on an option bit in the Configuration register in page zero of SAM compatible I/O space. Eventually, all PCI slots handled by the defective ASIC will be set broken. In the meantime corrupted I/O data is detected by checksums and handled by higher level protocols.

6.4.3.14 I/O Non-single-side Access, Different C-D Data

• fault site 8

• break via loopback on data

This fault deals with an internal I/O board fault that results in different data being driven out of each Gambit ASIC during a regular (not single-side) read retum. Each ASIC will disagree with the data driven by the other side and the board wilt break with a loopback on data error. The error sequence will be executed and the operation will be retried by the CPU and get noACKed.

6.4.3.15 I/O Board Open

• fault site 9

• break via loopback on control

This fault deals with an open, either from a broken etch or a lifted pin on the I/O board. The routing is very important for this class of fault The etch between the C-side and the D-side should be routed through the connector pin. This limits the possibility that an open on the I/O board is mistaken to be an open on the backplane. The driving ASIC sees no errors and the otherside ASIC asserts bus error; the board breaks the following phase with a loopback on control fault. After the error sequence the operation will be retried with no error and get noACKed.

6.4.3.16 I/O Board Short

• fault site 9 break via arbitrary broken

This fault deals with a short on the I/O board. When the fault occurred and who was driving the info bus during the fault are not relevant issues for this class of fault. During Postl of the transfer all ASICs on the bus will detect a bus error and then drive bus error during Post2. All ASICs on the bus signal a bus error during CPUpostand IOpostl. During Err2, the designated board will go broken based on the fact that it has detected bus errors during the error sequence and the other board did not go broken after its error sequence.

21 July 1995 37

?_ s

Xbus Functional Specification Stratus company Confidential

6.4.3.17 Transient Fault fault site anywhere action depending on fault site and timing

This fault deals with a transient fault. A transient fault is defined as a fault that is detected, but cannot be reproduced by the error sequence. If the fault is on the driving board and it is caught by loopback check, that board will go broken. If this is not the case then no board will break until Err2, then the designated board will go broken.

6.4.3.18 Slow Driver Fault

• fault site 4,9

• possibly break the wrong board with a loopback on control fault

When an ASIC with a marginally slow driver drives the bus, four ASICs clock in data trom the net while the net is changing. Due to differences in speed between ASICs from different tots, its possible that some of the ASICs will detect a bus error, and some won't, resulting in a loopback on control fault. This may result in one or both of the boards on the bus going broken.

6.4.3.19 Board Not Broken Generation

The boardjiot J>roken_out signal is generated by all boards. If a board is going to break the normal operation is for the board to assert bus error. All boards will enter the error sequence and the board that was going to break will de-assert board _not_broken jaut during the error sequence. It is possible that a board could break without asserting bus error. This could only happen if there is a problem with asserting bus error on the d-side gate array (for instance the clock line going to the flip-flop that produces the control signal (bus error) opens). Please refer to figure 50, Board Breaking Timing and latching (arrived at phase_l2_l) on page 89 for detailed broken timing diagrams.

6.4.3.20 Arbitrary Breaking

There are a series ot cases mentioned before that error logic cannot determine where the fault is. For these cases the goal is to not use the bus that the two boards are connected to. The default is to break the CPU board (designated board). This makes sense because there is a partner running in lock-step so no connectivity is lost. It is possible that the CPU board will break but the fault really lies with the PCIB. The only way this will pose a problem is if the failure in the PCIB effects the other bus that is still connected and is being used. Normally it is the CPU board that is the designated board, but if bit 21 of the Bus Interface State register is set then the PCIB board will be the designated board. THIS IS ONLY INTENDED FOR LAB DEBUG. UNDER NORMAL OPERATION THE DESIGNATED BOARD MUST BE THE CPU BOARD. IF THIS IS NOT ADHERED TO IT IS POSSIBLE TO HAVE BOTH BOARDS BREAK OR NO BOARDS BREAK.

21 July 1995 38

Xbus Functioi. . Specification Λratus Company Confidential

6.5 Summary Bus State Diagram

A flow chart for the basic bus operation described in the previous sections is shown in figure 12. The thinner state transition lines represent "normar' operation and the thicker lines represent exception conditions (for the sake of this diagram, losing arbitration is considered an exception case).

Figure 12. BASIC Bus Operation Flow Chart

21 July 1995 39

^^

Xbus Functional Specification Stratu . mpany Confidential

6.6 Arbitration

^" V

The( us arbitration mechanisnjon Polo is the key to enforcing the synchronization of the four buses in the Polo architecture. Th arbitration algortthrr maintains the appearance of one functional bus. This is accomplished through a series of point to point request lines that are part of the control buses. Figure 13 shows the interconnect for the arbitration network.

Figure 13. Inter-CPU Bus Arbitration Network

This arbitration network is a break from previous Stratus systems. Unlike the binary search arbitration network used on Jetta and Stratebus based systems. Polo uses a distributed arbitration based on the request signals that each board dπves.

As tn Jette, the determination of a bus grant depends on more than the request lines. Bus Grant can be inhibited by any board about to enter the data phase of a block transfer, or by an error on the bus. Thus any arbitration phase where grant inhibit or a bus error is asserted will not result in a grant for any board. The grant inhibit interconnect is also shown in Figure 13. Grant inhibit is also part of the control bus. A board dπvtng grant inhibit will dπve all three sets of its grant inhibit lines. This guarantees that all boards see the same grant inhibit and will make a correct arbitration decision.The grntjnh signal is the mechanism used to guarantee that a block transfer will occur in successive clock ticks, grntjnh is also use as the mechanism to enforce a dead cycle on the

21 July 1995 Z \ 40

Xbus Functiont-- specification atus Company Confidential

Xbus to turn the bus around. A bus master will always drive gmtjnh when it is actively dnving the bus. This guarantees that no device will be granted the Xbus the cycle after someone has driven info or date on the Xbus. A functional timing diagram which illustrates the use of grntjnh is shown in figure 15, High Level Block Transfer on page 45.

Table 6. Arbitration Priority Rotation

Table 6 shows the sequence of arbitration addresses that a board will follow when rotating if s address pnoπty. Arbitration value zero has the highest pπonty and arbitration value 3 has the lowest pnoπty. The sequence is such that adjacent boards are always assigned partner addresses, and that all boards will have each pπonty level at one time or another. For example, in a four slot system where all boards rotate arbitration priority together, every board will have the highest prioπty 1/4th of the time, the lowest priority 1/4th of the time, and each priority in between for (a total of) 2/4ths of the time in every cycle of address rotation.

The system maintenance interrupt signal is used to reset the rotating arbitration address on each board to the board's slot id. For example, when a new board is inserted into the system, all boards must have their arbitration addresses reset to be their slot addresses, since the new board has no means of knowing what its arbitration address must be other than its own slot address. When the progression through the addresses shown in Table 6 resumes, all boards will continue to arbitrate with unique arbitration addresses. Note that the assertion of maintenance interrupt has no effect on ttie actual arbitration period, therefore a node's priority at tiie beginning of the arb penod may be different than at the end due to a maintenance interrupt. However, every node will always modify their arbitration prioπty, either from a maintenance interrupt reset or an end or arb period increment, synchronously on the same clock edge. Since a maintenance interrupt is issued when a board leaves reset, it is guaranteed that no board will try to arbitrate before it's arbitration address has been synchronized with the remainder of the system.

Note. There are other rotation sequences that could be argued to be "more fair" in that the pπority will jump from a high pnonty to a low priority instead of progressing within a grouping of either high or low values, but in an effort to maintain simplicity this rotation scheme has been chosen.

The duplicate slot id signals on each board are latched at reset inside each ASIC and venfied that they are identical. A miscompare between the two signals will break the board while it is in reset.

The actual arbitration address may be implemented with a simple two bit up-counter that is XORed with the slot id. This counter will increment at the end of each arbitration penod on the bus. An arbitration penod is the time when at least one board has a busjeq held active. A board can only win the bus once duπng any arbitration penod. To clarify: some number of boards are arbitrating for the bus, with one board winning. The losing boards re-arbitrate, with one of those winning. This process repeats itself until all boards win arbitration. During this whole time, at least one bus eq is asserted. Any board that wins arbitration cannot arbitrate again until all bus eq signals have been de-asserted. Any boards that haven't arbitrated during this period may join the remaining boards. If all busjeq signals have been negated then all boards in the system have had the

21 July 1995 41

2^ ^

Xbus Functional Specification Stratus mpany Confidential

chance to be bus master once during that arbitration period. The arbitration period is now over and arbitration priority rotates.

A board always arbitrates against the other three boards in the system. If the CPU boards are duplexed, the duplexed pair arbitrates simultaneously. If either of the CPU boards wins the arb, both act as though they have won it, and both drive their info on the bus. This means that the duplexed CPUs will win half roughly of the arb cycles, and each PCIB will win one quarter of the arb cycles. When the CPUs are simplexed, each board will win roughly one quarter of the cycles.

During a bus error operation, as described in section 6.4, Bus Errors, on page 30, all nodes will de- assert their request and grant inhibit control signals. This will insure that the priority remains in tect during Xbus error recovery.

Figure 14 illustrates a case where arb_req prevents a board from re-arbitrating until each board that is requesting the bus gets a chance to be the bus master. Assume here that there are only three boards in this system. In this case board 1 has the highest priority, board 2 has the next highest and board 3 has the lowest priority. All three boards begin to request the bus in the same cycle (although it should be noted that this is not a requirement for this type of situation to occur). Here board 1 wins the arbitration and its access is BUSYed, thus it must re-arbitrate for the bus but cannot until the current arbitration period has ended. Notice that at least one busjeq is held active until Board3 has been granted the bus, which guarantees that Board3 will get a chance to be busmaster before any other board returns to arbitrate.

21 July 1995 ^y^ 42

Xbus Functiona. pecincation .atus Company Confidential

Figure 14. Arbitration After Being Busied

Note that there is always a dead cycle between two different boards' info cycles. It is necessary to allow one cycle to turn the bus around to avoid unwanted bus fight.

6.6.1 Bus Mastership Upon Bus Grant

If a pair of duplexed boards arbitrate for the Xbus then when the higher priority peer wins the grant, both boards will assume mastership of the Xbus. However, only the higher priority board will actually drive the Xbus because only the Xbus lines driven by the board winning the grant are watched. This is due to the fact that a bus receiver cannot know whether or not the requesters are duplexed so it will not know if valid date will be present on the two buses sourced by the non- winning board. Therefore these two buses will be left tristated.

6.6.2 Special Arbitration Rules for Non-Paired Read Returns

A duplexed board whose partner gets a non-paired register read request must go through the motions of returning the date in order to remain lock-stepped. However the non-target duplexed board in a non-paired read retum must not arbitrate for the bus and instead wart for its partner to win the bus. According to Xbus only the buses attached to a board that won an arbitration phase are looked at. Only the target board of a non-paired access will drive bus request in order to guarantee that the correct data is looked at tor a non-paired read retum. The non-target board will

21 July 1995 , 43

Xbus Functiona ,l Speci ^•« fi„ca„♦t ^; i„o„n Stratus

continue to go through the motions on the read retum but it will not drive the bus since only the board winning arbitration needs to drive.

21 July 1995 44

Xbus Functioi . peci ucation cratus Company Confidential

6.7 Block Transfers

The Xbus permits boards to hold bus mastership for a block transfer via the grant inhibit (grntjnh) signal. This does not allow for multiple block transfers. A board wanting to perform a block transfer asserts the gmtjnh signal the arbitration cycle after it first wins arbitration just as if it was doing a single cycle transfer. Then the signal is activated for as many additional arbitration phases as needed for the block transfer. This signal gives the board highest priority regardless of other arbitration activity, thereby permitting it to retain bus mastership for additional transfers. Note that the gmtjnh signal is used for all cycles, not just block transfers.

Figure 15 below provides a high level conceptual diagram of a block transfer which retains bus mastership for 4 bus operations.

Figure 15. High Level Block Transfer

winning the subsequent arb phases.

6.7.1 Bus Errors

The Xbus block transfer error recovery mechanism is architected so that the same mechanism is applicable to block transfers of any length. The burden of error recovery is placed on the initiator of the transfer. All bus nodes will be responsible for de-asserting their request and grant inhibit control lines to insure that the arbitration priority remains in tact during the Xbus error operation. Figure 16 below illustrates what this means for a bus error during the first operation of a four operation block transfer. the master must retransmit any data associated with the block transfer maintaining correct order

21 July 1995 45

T3έ 0

Xbus Functional Specification Stratus ^ .npany Confidential

• The grtjnh signal must be reactivated in time to keep the entire block transfer contiguous

As discussed later in this section, there are also situations wherein a bus error causes a block transfer to be canceled, requiring it to later be restarted from scratch, i.e. requinng it to rearbrtrate for the bus.

Figure 16. Error During First Operation of a Quad Block Transfer

(The error phases are shown condensed to fit on the page)

Similarly, errors during the second, third, and fourth operation for a quad block transfer are illustrated in the figures that follow.

Figure 17. Error During Second Operation of a Quad Block Transfer

21 July 1995 46 ι

Xbus Functional Λus Company Confidential

Figure 18. Error During Third Operation of a Quad Block Transfer

Figure 19. Error During Fourth Operation of a Quad Block Transfer

The four cases in the preceding diagrams illustrated error recovery when an information cycle associated with a particular block transfer is ERRORed. Should a bus error occur which causes all the accesses associated with a block transfer to be aborted, that block transfer is effectively canceled. The initiator must rearbitrate for the bus and start from scratch. The net effect is that at any point in time, only one block transfer is using the error recovery protocol.

21 July 1995 47

Z ^' V«

Xbus Functional Specification Stratus company Confidential

This is illustrated in figure 20 below. Two 2-operation block transfers are shown. The second information transfer of the first transfer gets ERRORed. This block transfer still completes using the recovery mechanisms just discussed. All cycles associated with the second block transfer on the other hand are completely aborted as a result of the error. This block transfer is then completely canceled and its initiator at some later time must rearbitrate for the bus and restart the transfer. Had the second block transfer been long enough to have grant inhibit asserted at the time the first block transfer is ERRORed, it deasserts grtjnh upon seeing the bus ERROR.

Figure 20. Error causing entire block transfer to be canceled

6.7.2 Global Writes

Global write operations are a variation of peer-to-peer transactions. During the echo transaction of a global write, the PCIB clocks in data at the same it is sending data to the CPU. In this way, all boards in the system see the global write synchronously. It also means that only a CPU can issue global writes.

6.7.3 Board Breaking

It is possible for a simplexed board to be performing a block transfer and then break part of the way through the transfer, resulting in a never to be completed partial transfer. For this reason, all boards targeted by a block transfer must watch the TRIDs of the entire block transfer. Should the TRID change before a block transfer was supposed to have completed, the bus ASIC needs to pad the block with O's to replace the missing data.

Though padding the block with O's to replace the missing data may at first appear undesirable (no data is better than bad data), a few factors must be kept in mind:

1) This is only an issue for simplex I/O sources such as disk and ethemet The software for these boards must tolerate the board "disappearing" and hence must have recovery mechanisms for incomplete blocks (O's in the memory are no worse than whatever random information would have been there prior to being overwritten by the I/O board).

2) The Stratabus based systems exhibit the same characteristic.

3) To not do this has a potential performance impact: bus ASICs may have to otherwise hold the entire block prior to passing onto the board rather than passing good date as it is received.

21 July 1995 48

Xbus Functional deification αtus Company Confidential

4) This situation should only occur on CPU initiated transfers. On the CPU, an LPMC is returned coincident with the read data.

Individual bus interface designs may choose to provide a notification mechanism should this occur.

It should be noted that the breaking board will drive a double bus error when it breaks. In the case described above, the errored information should get re-transmitted, but never will because the board has broken.

6.7.4 Bus Busys

A block transfer may only be BUSYed during the first phase of the transfer, i.e. BUSY may be asserted during the Post2 phase associated with the first information transfer. This single BUSY has the effect of canceling the entire block transfer, in practice, it is too late to prohibit the transfer of the subsequent sub-operations on the Xbus so the information in these sub-operations is simply ignored. The transfer initiator must re-arbitrate for the bus and repeat the block transfer.

Note that bus error takes precedence over busy. The only reason a subsequent cycle will ever have BUSY asserted is if a board misinterpreted the information within the cycle or is about to break. In both these cases, ERROR will be asserted concurrently with BUSY and should take precedence.

6.8 Peer to Peer transactions

Peer to peer transactions between the two CPUs are based on a simple store and forward protocol. A CPU sends a transaction on any of its two buses attached to a non-broken PCIB. Once any errors have been resolved (the Post2 phase of the last operation has passed without error), the PCIB(s) echo the transaction back up to the CPUs. The send and echo transaction are not spli the driving CPU holds grant Jnhibit until the complete send-echo pair is complete. This is to prevent re-ordering problems arising from eviction flush queue (EFQ) and read/write queue (RWQ) operations passing each other. Read transactions are split between the request and the date retum. An indivisible send-echo carries the read request. Other operations may intervene while the read is performed; then the data is returned in an indivisible send-echo data retum. The CPU initiator of the peer-to-peer operations must track the entire operation to see whether the cycle is acknowledged, busied, or errored.

Acknowledge and Busy are ignored for the first half (send portion) of a peer-to-peer transaction. A busy response on the echo will terminate the transaction and it will be retried at a later time beginning with the send portion. The send cannot be busied. The echoing board needs to track the error line during the echo so that an error on the echo portion will cause the normal seven phase error protocol followed by a re-echo of the transaction starting from the first phase that was errored and then proceeding forward. The send portion is not repeated in the event of an echo error.

The initiating CPU needs to keep track of whether a cycle will be peer-to-peer or not and indicate it on the Xbus via bit 4 of the second half TRID. Table 7 lists the Xbus operations that the CPU will mark as peer-to-peer by using bit 4 of the second half TRID. The cycles listed under non peer-to- peer operations will not be echoed. At a minimum the peer-to-peer initiator must drive grant inhibit from the first info phase of the send until one phase prior to the first info phase of the echo. The board performing the echo must handle grant inhibit from the first phase of the echo until the completion of the cycle. The PCIB will only echo operations marked as peer-to-peer via the TRID

21 July 1995 49

2

Xbus Functional Specification Stratus >mpany Confidential

bit The peer-to-peer bit in the TRID is only set for the send, the echo sets it back to zero since a Table 7. Peer to Peer and Non Peer-to-Peer Operations

a. bit 22 of the Bus interface State Register cycle with this bit set will not be responded to.

Only CPU's can issue peer to peer transactions, because only PCIBs have the buffering to perform a send and echo. PCI peer to peer transactions are not supported, and all traffic between PCI cards must occur via main memory.

Note: the following diagrams show AO, BO, A1, B1 info buses, not an idealized overlaying of operations on a single bus (as in the preceding).

21 Jufy 1995 50

QΛ

Xbus Functional icifiwition tus Company Confidential

Figure 21. Peer to Peer Cycle Initiated by a Simplexed CPU

Figure 22. Peer to Peer Non-paired Read from a Duplexed CPU

PCIB drives gmtjnh PCIB drives gmtjnh

6.8.1 Bus Errors

A bus error during the send portion of a peer-to-peer cycle is handle identically to any other cycle. Figure 23 illustrates this. A bus error in the echo portion will result in execution of the error sequence followed by the remainder of the echo transmission.

21 July 1995 2Λ 7- 51

9 81

Xbus Functional Specification Stratus npany Confidential

Figure 23. Store and Forward with Error on 2nd Operation of Quad Block Transfer

21 July 1995 52

Xbus Function Specnication tratus Company Confidential

6.9 Major Transactions

There are three types of transactions that can be performed on the Xbus, (non-split) memory writes, (split) memory reads and (split) flush/purge type transactions.

Non-split transactions are when the bus master does not require any data or response from the slave device to complete the transaction. These transactions supply an address in the first operation and the data in the subsequent sub-operation(s). All data write transactions are non-split and use this protocol.

A split transaction is 'split' into two separate bus operations, a request followed by the response. A block read is a good example of this. The master initiates this transaction by a read request for X bits of data tor a given address (the function code, virtual and physical address are all supplied in the info phase). The slave who has decoded the function code and address then performs the request on-board. When it gets the data, it returns the data via another bus operation. A transaction id (TRID) is used to match requests with responses and is unique to all transactions pending.

Flush and purge transactions are split. They are unique only because they do not need to see a bus operation returning date to complete the transaction. Any flush or purge transmitted from Cougar to Cyclops is transmitted on the Xbus. It is normally just looped back on the Xbus. However, if the board is in update mode, the transaction is performed as a peer-to-peer transaction. They will always see an INFO RETURN FUNCjDP (or INFO RETURN 256 when returning data) as the retum on this transaction. The virtual index and physical address is included in this type of retum.

Notes:

• The figures in this section contain a horizontal line that indicates either a tri-state or dont care condition.

» All signals are drawn active high so they are 'HI' when signals are in their TRUE state.

• bus eq and gmtjnh are driven on both cycles of the ARB phase. Note they are clocked in on the first half of the ARB phase and evaluated for the grant during the second half of the ARB phase.

• When VADR is seen in these figures, it actually represents the function code and virtual index. See section 6.10.2 on page 63 more details and bit definitions.

• When the note 'OPIN' appears on the TRID signals, it represents operation information and transaction defined information that will be passed with the bus operation, see section 6.10.1 on page 62 for more details and bit definitions.

• The Xbus control bus bits have been spilt out for readability. For example, BUS_REQ and ACK are drawn as two separate signals while in reality they are both on control[0]. Likewise BUSY, ERROR, and GRANTJNH are broken out from the control bus (maintjnt is not shown in these diagrams).

6.9.1 Write Transactions

A write transaction is completed in one bus operation with from one to four sub-operations for the date transfer. The initial bus operation supplies the function code, the virtual address, and the physical address in the info phase. The accompanying sub operation(s) are used to supply the data needed for the transaction.

Figure 24 Illustrates a 32 bit write transaction. Note it is up to the bus master to assert GRANTJNH (after receiving a grant) to reserve the info phases for the date transfer(s).

21 July 1995 ' ,^ ι 53

Xbus Functional Specification Stratus .npany Confidential

Figure 24. 32 Bit Write Transaction

^* See notes on page 53.

A block write transaction is much like a 32 bit write operation. The only difference is that the master must assert GRNTJNH for additional phases to reserve the extra info phases needed to transfer the extra data. Figure 25 illustrates a block write transaction.

Figure 25. 256 Bit Block Write Transaction

'See notes on page 53.

6.9.2 Read Transactions

A read transaction requires two bus operations: a request operation followed by the response operation. These two bus operations will always have the same TRID. The request operation does not have any accompanying sub-operations, but the response operations wilt require sub-

21 July 1995 54

2 A S

Xbus Functions _^ peciu-ation atus Company Confidential

operations if the amount of data requested cannot be transferred in one info phase. Figure 26 illustrates a basic read transaction with no sub-operations.

Figure 26. 32 Bit Read Transaction

The slave then responds with the data as follows: Figure 27. Slave Response to a 32 Bit Read

^* See notes on page 53.

Block read transactions are transactions that requires sub-operations on the data retum operation. Figure 28 illustrates this type of transaction.

21 July 1995 55 l ^ fø

Xbus Functional Specification Stratus company Confidential

Figure 28. 256 Bit Read Transaction

*See notes on page 53.

The slave then responds with the following.

Figure 29. Slave Response to a 256 Bit Read Transaction

6.9.3 Flush and Purge Transactions

Flush and purge operations look just like read transactions with the exception that the response may not have data to retum. The retum for these operations will always have FUNC_OP true with an INFORMATION RETURN (or INFORMATION RETURN 256) function code. The INFORMATION RETURN will have at least one bit of transaction defined information to pass back to the requesting device. The information that is passed here will be hit/miss information on the other board's cache. This information is passed during the second cycle of the info phase and uses the trid[6:0] bus lines (OPIN) (see section 6.10.1 on page 62). If returning date, it will use an INFORMATION RETURN 256 function code. Note that the virtual index and physical address are

21 July 1995 56

2

Xbus Functional jcifi -tion us Company Confidential

always passed back on the retum operation for these transactions. figure 30, Flush with Modified Date Transaction on page 57 and figure 31 , Slave Response to a Flush with Modified Data Transaction on page 57 are examples of a flush operation with a mod- data retum. Note this return will always be a cache line and that the return looks just like a cache- line write operation in that it has FUNCJDP true and supplies the virtual and physical address.

Figure 30. Flush with Modified Data Transaction

The slave then responds with the following. Note the state of the FUNCJDP signal. Figure 31. Slave Response to a Flush with Modified Data Transaction

^* See notes on page 53.

Figure 32 illustrates a flush transaction with no modified date to retum. Note that the retum for this transaction uses the info return FUNCJDP as a response. The TLB purge, data purge and instruction flush transactions look exactly like this transaction.

21 July 1995 57 τ i

Xbus Functional Specification Stratus npany Confidential

Figure 32. Flush (no modified data) or Purge Transaction

The slave then responds with the following. Figure 33. Slave Response to a Flush (no modified data) or Purge Transaction

'See notes on page 53.

6.9.4 Load/Clear Transactions

Load Clear Xbus transactions are hardware supported lock operations. A lock is won by performing a 32-bit load transaction in which the date received is non-zero. A zero result indicates that the lock is not won. A lock is cleared by issuing a clear transaction or by simply writing non¬ zero date into the memory lock location.

Load/clear transactions look exactly like a 32 bit read transaction. The only difference other than the function code is that the byte enables are not used. For more infomiation on load/clear transactions, refer to the Mercury Functional Specification.

21 July 1995 I ^Ά 58

Xbus Functional όpecitication .ratus Company Confidential

6.9.5 Ping Transaction

A PING transaction is a single bus operation transaction. Only the TRID is needed in this transaction. The only reason a PING transaction is done is to see if someone drives the ACK signal in POST2 of the bus operation.

6.9.5.1 The Need for PINGs

Because the Xbus uses split transactions, there is a need for the ability to handle cases where simplexed boards break and take themselves off-line while there is still a transaction pending on them. This can result in a device waiting forever for a TRID retum that will never come. This condition cannot occur on write operations because they are not split. See section 6.7.3 on page 48 for information on boards breaking during block transfers.

The system will crash when an on-line simplexed CPU/memory breaks so there is no concern with transactions that can only be directed to on-line CPU/memory boards. When CPU boards are off¬ line testing, there is a need to do I/O reads to them. This is a case where the testing CPU/memory can break and not cause a system crash but still leave transactions incomplete.

I/O boards can be simplexed while on-line or off-line and can break at any time causing the same conditions. The only split transaction directed to I/O boards are I/O reads.

Therefore, the list of transactions that has the potential of causing this condition is limited to I/O reads. I/O reads will require a test operation (called a PING) to gain infoπnation on whether or not a given TRID is still being worked on, i.e. if a board has broken or not.

6.9.5.2 Detailed Rules During PING Operations.

After the transaction master sends an I/O read operation it will start a timer. If the timer times-out before the TRID is returned, the transaction master will send a PING operation. This PING operation only conteins a valid TRID (the info lines must be deterministic but are considered a don care). The transaction slave device that is currently active working on this TRID will ACK the PING operation. The transaction master will then set the timer again and continue waiting for the TRID retum. If the PING is not ACKed, the transaction master will drop the transaction and assume the transaction has failed and/or the board has broken.

The slave device must keep a copy of all the I/O read TRIDs outstanding on the board. This is so that it may compare the TRID of all PING operations and ACK the PING when the TRID matches.

• The transaction master must ignore results of PI NGs if the matching TRID was marked complete. This can occur if the PING and TRID return were both on the bus at the same time (or if due to BUSYs that passed each other on the bus)

• The transaction master should mark TRIDs complete on the phase after POST2 and no-error of the TRID retum. If the TRID was complete at the exact instant the timer timed-out the PING operation can be sent, but if it is sent, the results of the PING must be ignored.

• The transaction master must not start any additional bus transactions on the same transaction master if a PING transaction (operation) is pending. This is so that an old PING is not using the same TRID as a new transaction. This has the potential of occurring when a TRID retum was seen before the PING completed it's bus operation.

• The slave must mark a TRID 'working' and stert ACKing PINGs on the Xbus before the transaction master timer times-out for the first time. There is ample time but the slave should do this as soon as possible. The cycle after POST2 and no-error of the request is the earliest this could be done.

21 July 1995 ~~ 59

Xbus Functional Specification Stratus mpany Confidential

* The slave should invalidate a 'working TRID' the phase after POST2 and no-error of a TRID retum. It could be done later but this only adds complexity. Note that in theory there may be only tour phases between POST2 of the retum and POST2 of a new request with the same TRID.

Figure 34. PING Transaction

If ACK was false, the transaction master will abort the transaction and formulate a ratum of O's with a tatted >p flag.

Figure 35. PING Transactions, Boundary Condition 1

21 July 1995 60

Xbus Functional _^ ,/ecitication tus Company Confidential

Figure 36. PING Transactions, Boundary Condition 2.

nothing. Thla Is true even It ping was not ACKed.

^* See notes on page 53.

6.9.6 busy, ack and error combinations

There are eight possible combinations of busy, error and ack on the xbus. The table below shows the outcome ot the bus operation for the given combinations of busy, ack and error.

Table 8. busy, ack and error combinations.

21 Jury 1995 61 ^r -

Xbus Functional Specification Stratus .npany Confidential

6.10 Xbus Signal Formats

6.10.1 trld[6:0] and func_ p__ Bus Format

A master that has won arbitration will drive these lines along with the info lines. Below is a list of signals internal to the ASIC that are multiplexed and transferred on the trid and tunc_op_ Xbus signals. trid[6:0] These are the transaction identification bits. failed_op This bit is valid during the second bus cycle of an info phase. It is only true during a peer-to-peer echo resulting from a failed peer-to-peer send. The info for phases that were transmitted without error are sent normally. The info for the phases that were never sent is set to zero and ttie failed_op bit is set indicating that this date was never transmitted so it has been zero-f ified. first_op This bit is only true during the first bus operation of a transfer of infomiation, i.e. it is only false during sub-operation and idle phases. Note it must be true during the first bus operation of a data retum. fuπc_op_ This bit when true signifies that the info lines carry a valid function code, address, etc. on this phase.

Hit Miss This bit represents the hit/miss status on special cache operations. It is zero on all other transactions.

IOVA This bit indicates that the address is an I/O Virtual Address and must be translated. peer-to-peer This signal indicates that the CPU bus master wants this current cycle to be driven peer-to-peer and it will control the grant-inhibit line to allow for it. This bit only has meaning when driven by a CPU and should otherwise be zero. remain[2:0] These bits represent a binary number that indicates how many bus sub- operations remain to complete this transfer of information (after this one). This field is zero for transfers that do not have sub-operations.

The figure below describes the format of the TRID bits (trid[6:0]) and the func_op bit on the first cycle of any INFO phase.

Figure 37. trid[6:0] and func_op_ Format for the 1 st Cycle of an Info Phase

func_op_

func_op_

21 July 1995 62

Xbus Functional v. . citiυdtion us Company Confidential

Figure 38 shows how the unique TRID filed is generated on the PCIB board in order to be able to check the PCI slot field in the I/O map RAM look-up (refer to Polo Programming Guide, section 5.4.1 , System Address Generation, for details on this checking).

Figure 38. trid[6:4] Format for a PCIB for the 1st Cycle of an Info Phase

The figure below describes the format of the TRID bits (trid[6:0]) and the func_op bit on the second cycle of any Info phase. The unused bits should be set to zero.

Figure 39. trid[6:0] and func_op_ Format for the 2nd cycle of an Info phase

func_op_ - remain[2:0] -a> failed op unused

Hit/Miss peer-to-peer first op

6.10.2 Info Bus Format

The figure below describes the format of the 32 bit info bus (info[31:00]) during the INFO phase of a function code operation (FUNC_OP). This is an illustration of an operation using a CPU address

21 July 1995 63

2 <

Xbus Functional Speculation Stratum company Confidential

Figure 40. Info[31:0] Format of CPU Address FUNCJDP

second cycle:

The figure below describes the format of the info bus for the INFO phase during a function code operation (FUNCJDP) when using an IOVA (I/O Virtual Address). Note that the RC bit field is zero; the C bit is stored in the map RAM and looked up during the IOVA to CPU address translation, the R bit is always 0 since there is no remote CPU board.

Figure 41. Info[31:0] Format of IOVA Address Bus FUNCJDP first cycle:

313029282726252423222120191817 16 15 14 1312 11 10 09 08 !07| 060504030201 00

-►!-« — ι-»|o o|^|^ — j 00 chunk device index checksum page func[5:0] byte_en[3:0]

second cycle:

21 July 1995 64

2- X

Xbus Function ltus Company Confidential

The figure below describes tiie format of the info bus on the first and second cycle of a of an I FO phase when the FUNC.OP signal is false (i.e. data). It is an example of an operation returning 64 bits of data. If the operation was returning 32 bits of data, the same data would be driven first and second cycle.

Figure 42. Info[31:0] Format During Data Transfers

first cycle:

second cycle:

31 30292827262524232221 2019 1817 161514 1312 11 10090807060504030201 00

6.10.2.1 Function Code (re and byte enable) Transfer

The function code bits are used to describe the Xbus operation and/or request. A function code is not passed on a normal date retum operation.

The function code bits are transferred on the first cycle of the Info phase. These bits are valid only when the FUNC_OP signal is true. func[5:0] use bus lines info[11:6] respectively. Below is a list of the basic operations that can be performed on the Xbus. These are defined within func[5:3]. Note that func[2:0] are re-defined for each basic operation however, the meaning of these bite were kept consistent where possible.

Note that table 9 through table 14 are subsets of the Golfbus function code definitions.

32 bit read operations func[5:3] = 000

Included in this group are all the read operations requests that only require a single 32 bit retum.

21 July 1995 65

2

Xbus Functional Speciti -ation Stratus -mpany Confidential

func[2:0] These bits represent a specific 32 bit read operation.

Table 9. 32 BIT READ OPERATIONS/REQUESTS

32 bit write operations func[5:3] = 001

Included in this group are all the 32 bit write operations transactions. func[2:0] These bits represent a specific 32 bit write operation.

Table 10. 32 BIT WRITE OPERATIONS/TRANSACTIONS

Special Cache Line Operations func[5:3] = 010

This group of operations perform special tasks on cache lines.

21 July 1995 2S > 66

Xbus Functional tus Company Confidential

func[2:0] These bits uniquely identify a bus operation

Table 11. SPECIAL CACHE LINE OPERATIONS

Read Retum Function Codes func[5;3] = 011

This group of function codes is used only internal to the bus ASICs. These function codes will never actually get transmitted on the Xbus. f unc[2:0] These bits represent how many data words are being returned.

Table 12. READ RETURN FUNCTION CODES

a. although the Xbus is 32 bits wide this function code is supported for backwards compatibility with the Cougar ASIC.

Block Memory Read Operations/Requests func[5:3] = 100

This group of operations are used to read blocks of data from memory. func[2] This bit is always zero

21 July 1995 67

Z l

Xbus Functional Specification Stratus company Confidential

func[1:0] These bits represent the number of info phases required to transfer the data on this bus transaction.

Table 13. BLOCK MEMORY READ OPERATIONS/REQUESTS

a. although the Xbus is 32 bits wide this function code is supported for backwards compatibility with the Cougar ASIC.

Block Memory Write Operations/Transactions func[5:3] = 101

This group of operations are used to write blocks of date to memory. func[2] This bit is always zero. func[1 :0] These bits represent the number of info phases required to transfer the data on this bus transaction.

Table 14. BLOCK MEMORY WRITE OPERATIONS/REQUESTS

a. although the Xbus is 32 bits wide this function code is supported for backwards compatibility with the Cougar ASIC.

Reserved func[5:0] = 11 xxxx

21 July 1995 68

Xbus Functioni. .pecincation atus Company Confidential

6.10.2.2 Remote/Coherent bits (re [1:0])

These bits were developed to support a MESI CPU caching scheme and the Golf CPU/memory architecture. They give the flexibility to optimize performance and to maintain cache coherency between all boards in the system. The meaning of these bits for a given basic operation is shown below.

For all memory read operations (block or byte specific), rc[1:0] represent the following.

00 /incoherent - Dont snoop, don't set remote tags. This will be used by CPU code fetches and reads to locations where cache coherency is controlled by software.

01 /coherent - Snoop but dont set remote tag. This will be used by boards without a cache when reading cacheable locations.

10 /remote/shared - Snoop, set remote tag. An example is a CPU reading shared data.

11 /remote/exclusive - Snoop, set remote tag. Infomiation will be passed to the CPU that exclusive rights are required on this cache line. A CPU (a board with a write into cache) will be the only board to initiate this type of operation.

For all memory write operations block or byte specific), rc[1:0] represent the following.

00 /incoherent - Dont snoop this, write this to main memory. This will be used on writes to locations where cache coherency is controlled by software.

01 /coherent - Snoop, invalidate and writeback any modified data, Dont change remote teg status. (Note: If a writeback is generated from a snoop of this type of write. The writeback will use the remote/exclusive protocol and that will clear the remote tag.) This will be used by I/O board writes that need to maintein cache coherency.

10 INVALID.

11 /remote/exclusive - Snoop, clear the remote teg. This will be used on CPU cache line writebacks.

For all other operations, rc[1:0] have no meaning and will be = 0. This includes the following:

I/O WRITE OPERATIONS.

I/O READ OPERATIONS.

PURGE OPERATIONS.

FLUSH OPERATIONS.

BOTH INFO RETURN OPERATIONS.

6.10.2.3 Byte Enables (b te_en[3:0]).

The byte enables are valid on 32 bit read, 32 bit write, I/O read and I/O write operations. On all other operations, these bits will be false (=0). When these bits are set, It signifies that the operation is requesting/modifying the indicated bytes. Figure 43 identifies each byte and its corresponding byte enable bit.

21 Jufy 1995 69

? £ ?

Xbus Functional Specification Stratus npany Confidential

Figure 43. byte_en[3:0] definition

Increasing byte _fcι aααress ^"

byte_en[3] byte_en[2] byte_en[1] byte_en[0]

Points to this byte points to this byte Points to this byte Points to this byte

By asserting 1, 2, 3 or all the bytes enables, all 8. 16, 24 and 32 bit operations are possible. This is shown below.

8 bit operations

3210 1000 Request or modify data[3124] (address[1 :0]=0) 0100 Request or modify data[23:16] (address[1 Λ]=1 ) 0010 Request or modify data[15:08] (address[1 :0]»2) 0001 Request or modify data[07:00] (address[1 :0]«3)

16 bit operations

3210 1100 Request or modify data[3i :16] (address(1 :0] _* 0) 0110 Request or modify data[23:08] (addressh :0]=1) 0011 Request or modify data[15:00] (address! :0]*2)

24 bit operations.

3210 1110 Request or modify data[31:08] (address[ :0]«0) 0111 Request or modify data[23:00] (address[1 :0]-1)

32 bit operation

3210

1111 Request or modify data[3l :00](address[l :0]»0)

6.11 Board Synchronization/Board States

Board synchronization is covered in more deteil in the ASIC and board specifications for the desired board. This section in the Xbus specification is limited to the board states necessary to perform the task of putting a board on-line simplexed or on-line duplexed. It gives a deteiled description of the type of Xbus transactions that are possible in each state.

In Golfbus based systems, a mode known as simplexed mode was implemented for certain debug situations. This mode was never used or tested, and it has been removed on Polo.

21 July 1995 70

2

Xbus Functional _ _f jecitιcation jtus Company Confidential

6.11.1 CPU/Memory Bus Interface Modes

Below is a description of the board states needed to sync two CPU/memory boards (i. e. bringing a CPU/memory board on-line-duplexed). This is considered a super-set of all the modes necessary for booting and/or bringing a CPU/memory board on-line-simplexed.

The following signals are required on this type of board.

Broken, On-line, Duplexed, Update, Freeze-stete (BODUF) (simplexed.mode bit false)

BODMUF 1XXXX Broken: Board is invisible to the Xbus. The board will ignore all Xbus operations. The board must be reset through the dedicated reset lines before it can be accessed at all.

00000 Off-line: The board will respond to non-paired I/O read cycles and will perform non-paired and Global write transactions. It will ignore all other bus operations. The board will arbitrate for the bus to retum I/O read date. The board is capable of initiating bus transactions but conceptually, the board should not generate transactions. Software should enforce this where needed.

00010 Off-line/Update: The board will respond to non-paired I/O read cycles and will perform all I/O write transactions (Global, Paired and non-paired). The board will NOT respond to paired I/O read transactions. The board will also perform all memory write transactions that index into its memory array. The board is capable of initiating bus transaction and will generate only odd TRIDs regardless of what slot it is in.

01000 On-line: This is the normal state of a board that is on-line simplexed. The board will listen to and perform ALL bus transactions directed to him. The board is capable of initiating bus transaction and will generate only even TRIDs regardless of what slot the board is in. In this mode, the board will respond to and drive the Xbus during a retum of I/O paired read data.

01010 On-line Update: This is the state of a simplexed on-line board that will broadcast all local memory writes to the Xbus. The board will listen to ALL bus transactions directed toward it. The board is capable of initiating bus transaction and will generate only even TRIDs regardless of what slot the board is in. In this mode, the board will respond to and drive the Xbus during a retum of I/O paired read date.

01011 On-line/Update/Freeze: This is the state of an on-line simplexed board when it is doing the final copy of state over to it's partner board. The board will ONLY listen to transactions that have the same TRID as its slot number or its partners slot number, (i.e. odd or even TRIDs). The board will BUSY all other new transactions directed towards him. The board may initiate transactions and will use even TRIDs.

00011 Off-line Update/Freeze: This is the state of an off-line board that is getting state copied in via I/O space. The board will listen ONLY to transactions that have the same TRID as if s slot number or ifs partners slot number, (i.e. odd or even TRIDs). The board will BUSY alt other new transactions directed towards him. The board may initiate transactions and will use odd TRIDs.

01100 Duplexed: Entered only after the board went through the Sync point and it was a duplex request Both boards enter this mode simultaneously and are considered duplexed and running in lock-step. The 'update' and freeze-stete' bits are cleared upon entering this mode. Odd and even slot boards will generate even TRIDs. On paired I/O reads, the board will perform the read of the register in lock step with it's partner, and both boards will drive the Xbus during the retum of the data.This mode is cleared in the result of a T/A failure, or by this board or its partner going Broken.

21 July 1995 71 ecz

Xbus Functional Specification Stratus pany Confidential

Online for Dumping, loop-back mode: These modes are unsupported on the Xbus.

Note: States not listed are considered illegal

Table 15 below gives greater detail on operations performed. Table 15. CPU/Memory Board States ³

a. KEY 1 State bit true 0 State bit false. X State bit true or false P The board will perform this Xbus transaction fully. M The board will go through the motions of performing this Xbus transaction but will not dnve the Xbus in the info phase of the data retum for this transaction.

I The board will ignore this operation. (1 e. will not dnve ACK, BUSY or retum data B The board will busy this transaction unless the TRID of the transaction matches its slot number or its partners slot number. Note: the board will busy the transactions until it is released from this mode EFQ Freeze State busies cycles sent to the EFQ (except data retums), and Freeze Sute busies cycles sent to the R Q. Refer to section 3.3 for more details on freeze states, the EFQ. and the R WQ

N The board will never initiate a transaction in this mode. O The board can initiate all transactions in this mode and will only use odd TRIDs E The board can inmate all transactions in this mode and will only use even TRIDs b Refers to either EFQ Freeze State or Freeze State c CPU I/O wnte refers to an I O wnte that is broadcast to all CPUs.

21 July 1995 72

O

Xbus Functional <JCiti _tion S. .us Company Confidential

Figure 44 shows the recommended state transitions for CPU/memory boards. Not all possible transitions are shown here, only the ones necessary for software to put a board on-line simplexed or on-line duplexed. A board can transition to the broken state from any state but this is not shown in the diagram to improve readability. The dotted line is the fast duplex path.

Figure 44. CPU/Memory Board state transitions

6.11.2 Simplexed I/O Board Bus Interface Modes

Below is a description of the board states needed to bring a I/O board on-line that has no capability of duplexing. Note, a simplexed I/O board will never respond to (or ACK) paired I/O read transactions.

The following signals are required on this type of board.

Broken On-line (BO) BO

1X Broken The board is invisible to the Xbus. The board will only perform I/O writes when possible. If the board is busy performing an I/O transaction, the board will attempt to busy additional I/O transactions. Because the board is broken, it cannot drive the busy Xbus signal. This means that any I/O writes that occur when a broken board is busy fall to the bit bucket and are not performed. The wnte will also not occur If defective hardware precludes the write from occurring.

21 July 1995 73 >

781

Xbus Functional Specification Stratum o pany Confidential

00 Off-line The board only responds to non-paired I/O read and I/O write transactions. The board will arbitrate for the bus to retum I/O read data. This board will never respond to (or ACK) any paired I/O read transactions. The board is capable of initiating bus transactions but conceptually, the board should not generate transactions. Software should enforce this where needed.

01 On-Line/Simplexed This is the normal state of a board that is on-line simplexed. The board will listen to ALL bus transactions. The board is capable of initiating bus transaction and will generate TRIDs that are equal to its slot ID. This board will never respond to (or ACK) paired I/O read transactions. loop-back mode This mode is unsupported on the Xbus.

Table 16. Simplexed I/O Board states*

a. KEY. 1 Sute bit true. 0 State bit false. X State bit true or false T Will perform if possible (i.e. if not busy performing one just before it)

P The board will perform this Xbus transaction fully. I The board will ignore this operation, (i.e. it will not drive

ACK. BUSY or retum data.)

N The board will never initiate a transaction in this mode. Y The board can initiate all transactions in this mode and will use the TRID of its slot

21 July 1995 74

Xbus Function... .peciiication atus Company Confidential

Figure 45. Simplex I/O Board State Transitions

6.11.3 Loopback mode

The Xbus does not support Golfbus style loopback testing.

6.12 Board Breaking/Board Removal

6.12.1 Un-breaking a Board

Boards should generally be "unbroken" by resetting them rather than directly clearing the broken bit in the Bus Interface Status register (see the FTIO / PCIB Programmer's Guide). There is at least one (unlikely) scenario where simply clearing the broken bit in the Bus Interface Status register can cause a board to receive multiple read retums, specifically:

A board acknowledges an Xbus read request targeted to it and then breaks. While it is broken, the original requestor PINGs the read. Since the board is broken, the PING will not be ACKed and the requestor will then retum O's to its on-board processor. If software then simply clears the broken bit in the Bus Interface Status register, the original board may then retum the read to the requestor (assuming it took awhile to process the read), thereby giving the requestor a second response on a read it has already passed up O's to the processor for. Clearing broken via reset in this case would have ensured that the broken board clears the read response out of its state machines prior to unbreaking.

Also, there should be some delay between the time a board breaks and when it un-breaks and starts performing Xbus accesses to other boards. The following (extremely unlikely) danger otherwise exists:

A board performs a read request, then breaks, and then gets unbroken via a reset. At that time, it immediately generates another read request and happens to reuse the same trid. If s possible that the read data from the first (pre-broken) read (which may be to a completely unrelated register) then comes back and gets grabbed for the second (post-broken) read.

If s difficult to quantify how large of a gap is necessary between breaking and issuing the first read access after unbreaking, but even without precautions, this problem is very unlikely. Normal onboard diagnostics which are executed as part of the unbreaking code are probably long enough in duration so as make the above scenario impossible.

21 July 1995 z 1 fαt 75

Xbus Functional Specification Stratus -ompany Confidential

7. Xbus Routing and Xbus Interface Clocking

All infoπnation transferred across the XBus is synchronously clocked onto and off of the bus at 24MHz. The bused, control and TA signals are driven and received directiy by the bus ASICs. The voted signals are buffered by 26S10 transceivers. Clock are generated from a single source located on the backplane and distributed through ECL clock lines.

7.1 On-board Clock Generation

The Polo project reuses much of the Jetta System clock design. The Sentry ASIC receives an 8 MHz clock from the backplane, and generates a 48MHz clock and 4MHz phasor for the bus ASICs. The Jette 48MHz 0.8 micron DLL (delay locked loop) and active clock spine are used for distributing a single 48MHz clock on the ASICs.

7.1.1 Clock Phases

All flops in the Gambit and Cyclops ASICs are clocked by a single 48MHz clock spine. However, it is often necessary to have PCI and Xbus related logic behave as if clocked at 24MHz or 2MHz. To accomplish this, we define a group of pseudo-clocks that run at 24 and 12MHz, and have rising edges at all of the useful times.

0ns 20ns 40ns 60ns

dk48 rι_πjn_π_r rL_rι_π_j

Cik48_12_0 Clk48_12j Clk48_12_2 Clk48_12_3

phase_i2_p I÷L phase_12_l phase_l2_2 phase_l2_3

21 July 1995 76

Xbus Function^ opectπcation .atus Company Confidential

There is really only one clock in the ASICs, clk48. However, by controlling flops with the phase signals on the clock enables, it is possible to make it appear that the design has all of the clocks shown above. Phase_12_0 is used to create clk48_l2_0, etc. The flops themselves are a D flop with a mux on the input, where the mux selects between new data and O output. The phasors are connected to the select line on the mux.

7.2 Signal Routing

The bidirectional info buses are routed point to point between boards. On each board, separate traces run from the connector to each ASIC; this allows the error protocol to determine if a fault is in the backplane or in the board. Series resistors are used to control signal quality, and CMOS levels are used for increased noise margin and reduced susceptibility to cross-talk.

Figure 46. Routing of info lines

oen

The three way voted signals are routed as separate point to point connections. The reset_x__,y_,z_ signals are driven by the CPU boards, wire-ORed on the backplane, and received by all boards. The board_not_broken_ signal is driven by a single board and received by all boards. These signals are driven by normal 4mA drivers of the ASIC, and buffered by 26S10 transceivers. The voted signals are terminated with a pull-up and isolation diode on both the receiver and driver side. This means that it is possible for there to be one or two pull ups on the line. However, whenever the line is being used for active signalling, there are exactly two terminators, one at either end of the line.

21 July 1995 77

2 d ?

Xbus Functional Specification Stratus company Confidential

Figure 47. Routing of 3way voted lines

7.3 Clock Fault Tolerance Issues, Outbound Signals.

The Polo system must protect against failures of a clock chip on a board. There are two physical broken lines on a board, one with all clocking from the C-side clock chip and another with all clocking from the D-side clock chip.

The Xbus has all clocking of signals on and off the bus performed within the bus ASICs. If a side of the board loses its clock, the other side will detect date loopback errors and break the board. The three way voted signals are buffered by 26S10 transceivers, and are not susceptible to clocking failures.

In Polo the clock distribution is not triplicated. An analysis on clock line distribution was performed and it was calculated that due to the small number of clock loads system reliability was lowered by adding extra drivers to the system to perform the clock routing triplication.

See the Polo Clock Distribution Specification and the Sentry Clock Recovery Device Specification for a discussion of clock fault tolerance at a system level.

21 July 1995 78

Xbus Functional , jcificαtion S JS Company Confidential

8. Board Insertion and Removal

8.1 Hot Plugging

Polo is the first Stratus produce to interface CMOS ASICs directly to the backplane. As such, it is the first to address live insertion of CMOS parts.

If a CMOS part is supplied significant current through its I/O pins before its power grid has come

21 July 1995 79 z ^

Xbus Functional Specification Stratus C-. npany Confidential

9. Xbus Fault Tolerance

The goal tor fault tolerance on the Xbus is for no single point of failure to cause a system crash or allow a transaction to complete without transmitting the correct data. A single point of failure could be any component, any signal, or any pin on a component.

The Xbus fault tolerance scheme assumes that one of the components connected to the bus is always responsible for hard bus errors. To this end, the design has been simplified around the assumption that when an error occurs, an offending Field Replaceable Unit (FRU) will be removed from service. This extends to include all buses that the FRU is connected. Thus when an Xbus fault occurs the faulted bus service is removed taking at least one FRU with it.

Both sides of the CPU board drive signals to the bus and both sides of the board perform various checks to ensure that the board is functioning normally. This is identical to the Golfbus methodology. The PCIB board behaves differently; both sides of the board drive and check the bus when driving date from Xbus related IO registers, and when driving data originating on a simplexed PCI card, only one ASIC drives and checks the entire width of the bus.

^ φ ^

The Xbus signals are divided into three classes: info bus signals, control bus signals, and three- way voted signals. The fault tolerance methodologies are different for each of these classes.

9.1 Info Bus Protection

The info bus is protected by parity and loopback checking. However there is no A to B bus cross checking by anyone on the bus other than the bus master. When a PCIB is bus master only one of the two buses attached to a given CPU module will be active because the PCIBs do not duplex and therefore the bus from the PCIB that did not win arbitration must be idle. Likewise the PCIBs cannot know if two duplexed CPU modules are driving the bus or a single simplexed CPU has ownership (two simplexed CPUs could have started arbitration at the same instant). Thus receivers on the Xbus will only listen to the bus that is driven by the module that won arbitration.

9.1.1 Parity

Parity is generated and transmitted across each of the physical buses. The 32 bit bus, trid field, and func_op bits are all covered by a single parity bit. The purpose of the parity check is to protect against etches that open between the transmitter and receiver. Boards in the system check the transmitted parity that they receive with parity that they generate themselves.

A bus error is signaled on any parity error.The bus_err signals are also effected by other error checking that is described later in this document. In an effort to simplify these descriptions the reader may assume that the actual error signals are a logical OR'ing of the various outputs from different checking blocks.

Both the C and the D-sides of the boards will perform the check and compare results individually and decide on any action that must be taken. In the event that the two sides of a board do not agree upon the status of a bus cycle, the board will break in the following cycle in one of two ways.

• The D-side of the board detected an error and the C-side did not. In this case the D-side will drive a bus error and the C-side will disagree with what the D-side has done and break the board. Here the transaction is errored and repeated, but the broken board should no longer be checking and the transaction will complete.

• The C-side of the -board detected an errcr and the D-side did not. Here the C-side will once again break the board, but this time because it expected to see an error on the bus and didn't.

21 July 1995 80

Xbus Functional _ deification •tus Company Confidential

The transaction will complete normally and the bad board will be removed from service.

CPU boards compute parity in both C and D ASICs and break if a defect arises in one side's parity generators. Most of the transfers from a PCIB board, however, originate from a PCI bus connected to a single Gambit ASIC. When the Gambit drives this single sided data, it drives all the bits on the Xbus, and an error in the simplexed parity generator would hang the system. Fortunately, there is no need to duplicate the entire Gambit ASIC, only the parity generation section.

Figure 48. Self Checking Parity Logic,

A fault at site 1 causes erroneous date to be transmitted with correct parity. This will be detected by higher level checksums.

A fault at site 2 will cause the board to break with a date loopback fault.

A fault at site 3 will cause the board to break with a parity generator fault

A fault at site 4 or 5 will break the board with either a data loopback fault or a parity generator fault.

21 July 1995 81

Xbus Functional Specification Stratus ui npany Confidential

9.1.2 Loopback checking

Figure 49. Loopback Connectivity

On normal accesses (non-CD different, non-single sided), each side of the board drives half of each field to the bus and receives all of both busses. Thus there are two checks that each ASIC can do on the data returning from the bus.

1 : Does the returning data match the data that was driven from this ASIC. This will be known as the loopback drive check.

2: Does the returning data match the data that was driven by the other ASIC. This will be known as the loopback compare check.

Two error signals are generated inside of each ASIC for each physical bus that it is driving. In order to know the true state of the board each side of the board must know the results of the comparisons on the other side as well as the comparisons it has done. Thus two signals are passed in each direction, as shown in figure 49 above, so that the each side may correctly determine the state of the board.

Each ASIC will determine wether the data on the bus is faulted or not, as well as if this board is broken or not. Table 17 below shows what combinations of the loopback signals for each bus indicate that the board is broken.

Table 17. Board Broken Matrix for a Single Bus

21 July 1995 82 *

Xbus Functiona. itus Company Confidential

Table 17. Board Broken Matrix for a Single Bus

A double bus error will be asserted and the notjϋroken signal de-asserted if a condition exists that should break the board, tn addition a bus error will be asserted for any condition where a compare error of any type has occurred and the board has not broken. The bus error will only be on the bus while the error is detected.

9.1.2.1 Loopback on CD different accesses

When an ASIC drives date from a CD different register on the bus, it performs loopback checking only on the trid, func_op_, and parity lines. It asserts bus error if it sees an error, and will go broken according to table 17

9.1.2.2 Loopback on Single-side accesses

When a Gambit ASIC drives single-side date from the PCI it performs loopback checking on all of the bits; it does this only for the purpose of asserting bus error, not to see if it should go broken. It will not break according to table 17, because the cmp_err and drive.err signals from the otherside are invalid. However, if the driving ASIC sees a bus error, and the otherside ASIC does not (parity received good), the board will break due to a loopback on control fault.

9.1.3 Loop_ck_ops

Loop_ck_ops are used during the error sequence to make intelligent choices about the state of various boards and the buses that they are attached to. A loop check op is a cycle in which a board drives an alternating pattern of AA's and 55's on the backplane. The board evaluates the pattem and similar patterns driven by boards on the other end of the bus. Based on this information, bus state is altered.

21 July 1995 83 z ^

81

Xbus Functional Specification Stratus ^mpany Confidential

9.2 Control Bus Protection

The Xbus collects control signals into a series of point-to-point unidirectional signals protected by an ECC code. Each board drives a separate control bus to each of the other boards.

There are two completely independent types of checks performed:

1 : Every ASIC corrects for any single point failures (shorts, opens, dead drivers or receivers) with a single error correcting, double error detecting ECC code. Single bit failures cause a maintenance interrupt and are togged, but otherwise have no effect.

2: If a driving board detects a double bit error on the outgoing control signals it will attempt to assert bus error and then it will break in the error sequence (it will break regardless of its ability to drive bus error). If a receiving board detects a double bit error on any incoming control signals it will attempt to assert bus error then it will break in the error sequence. If the error was actually on one board and only one board saw the eπor then the correct board will break. If the error was such that both boards saw it then both boards will break. This is to limit the possibility that data gets corrupted.

9.3 Three-way Voted Signal Protection

Some broken and reset related control signals on the Xbus are triplicated and voted. The reset and board_not_broken_out signals are buffered through transceivers because they must be valid during power up or power down events. The CPU signal sync_out is buffered through transceivers to reduce ASIC pin count. The CPU signal my_online_out is sourced directiy by the ASIC as a cost reduction.

The D-side ASIC drives signals to the transceivers and to the C-side ASIC. The C-side of the board checks what was driven by the D-side with what it thinks should have been driven. If the C- side disagrees with the D-side the board breaks.

Each ASIC that receives a three-way voted signal will perform a majority voting algorithm on the three signals to determine the value. In normal operation all three lines will be the same. If there is a fault on one of the three lines (either on the backplane or on the board), two of the three lines still will agree.

When a voting error occurs, the device detecting the fault should send a maintenance interrupt and log which transceiver has the non-unanimous vote. The logged information is stored in the Voter Error Transceiver Stetus register(s) and will not be over-written until the status register is explicitly re-enabled by software.

9.4 Error Reporting

A number of errors can be detected by a board's bus interface logic:

1 : A loopback error that breaks the board. This specifically refers to loopback errors on the info, parity, TRID, and funcjop signals which are seen by one side of the board and not the ottier during bus cycles in which the board drove the bus.

2: A disagreement between the two-sides of a board on whether a three-way voted signal should be driven.

3: A parity error on the bus, either on a cycle where the local board was driving the bus or another board was driving the bus.

4: A loopback error on the bus. 21 July 1995 ~~ 84

Xbus Functional eciti dtion tus Company Confidential

5: A three way voter error in which not all three signals are in agreement.

6: Any bus error signaled by any board in the system

7: A thermal fault (high temperature detected on the board)

Some of the above errors may break a board (e.g. #1 and 2), some may cause a bus error to be generated (e.g. # 3, 4 and 6), and some have no effect on normal machine operation (e.g. #7). The Xbus interface conteins two classes of status registers to record information surrounding the above errors: Broken Status and Error Status. Broken Status registers are only activated when a board goes broken, and are designed to pinpoint the logic that set broken. Error Status registers are for reporting non-fatal errors such as faults on the backplane. All of the Error Stetus registers are updated simultaneously on any bus error signaled by the system, or a voter or thermal error on the board. Specifically, when all error reporting is enabled, any of the above errors cause:

1 : A maintenance interrupt to be generated 2: The type of error condition to be stored.

In the above case, subsequent error state recording and the resulting maintenance interrupts are disabled until explicitly re-enabled.

It is not always optimal that all the above errors trigger the error registers and generate a maintenance interrupt. The reporting of new errors can be effectively blocked by an excessive number of reports on a known error. Hence, it is possible to turn off error latching due to different sources:

1 : Perform error latching and maintenance interrupts on all errors

2: Inhibit eπor latching and maintenance inteπupts of incoming three-way voter errors

3: Inhibit error latching and maintenance interrupts on thermal failure related errors

The error latching disable bits inhibit those faults from triggering the error registers. However, if a disabled eπor is present and non-disabled error occurs, both errors will be recorded in the eπor registers.

Details on the operation of this eπor reporting circuitry are provided in the detailed IO register definition section of the Polo Programmer's Guide. The specific registers that implement the above functionality are the Bus Interface Fault Reporting Registers, the Bus Info Error Status Registers, the Misc Eπor Stetus Registers, the Bus Eπor Transceiver Stetus[1:0] Registers, and the Voter Eπor Transceiver Status Registers.

9.4.1 Producing Errors

The Cyclops and Gambit gate aπays have specific registers which allow software to produce various eπors on the Xbus. The registers are described in deteil in Polo Programmer's Guide. These registers allow the eπor logic to produce all of the eπor cases described in section 6.4.3, Fault Conditions.

The basic logic to produce an eπor consists of XOR gates on all the info, trid, func_op, parity, control lines going out and all the same lines coming in. One input on the XOR is the data line the other input is the eπor line and the output is the resultant date. If the eπor line is high the data coming out will be in error. If the error line is low then the data coming out will be the coπect data. If date is eπored only on the input or only on the output then the eπor will be in the form of a loopback error. If the data is eπored on the input and the output then the eπor will be in the form of a bus error since it will not be a loopback eπor.

21 July 1995 - 85

Xbus Functional Specification Stratus npany Confidential

Table 18 on page 86 shows all the error cases and the necessary settings from a hardware point of view to produce the eπor. 'receive eπor* means the XOR gate on the receive side is inverting, transmit error' means the XOR gate on the transmit side is inverting. The category describes if the eπor occurs only on a particular pattem match or if the eπor occurs just after the register is written. For many cases it would be possible to produce an eπor with different cases than those described below. The goal is to only produce the eπor in one manner.

Table 18. Error Settings

21 July 1995 86

Xbus Functiona jβctnoation rtus Company Confidential

9.4.2 Maintenance Interrupts

Each Xbus board generates maintenance interrupts whenever an event occurs that would be of interest to the M&D software. These events fall into three classes: broken/unbroken events, error events, and software commands. As described in the following section, broken/unbroken events include board resets, board insertions, and board removals.

9.4.2.1 Broken Events

Boards transitioning to or from the broken state generate a maintenance interrupt. A board can break due to a detected hardware failure, a cold or warm reset, a board removal, or a software set broken command. A board can unbreak due to a software command, a reset or an insertion.

Cold or warm resetting a board causes that board to go broken at the start of the reset and unbreak itself at the end of the reset, thereby generating two maintenance interrupts assuming the board was not initially broken (one maintenance inteπupt if the board was initially broken). Inserting a board into the system generates a single maintenance inteπupt when that board unbreaks itself.

9.4.2.2 Error Events

A number of eπor events may cause maintenance interrupts. These are reported in the Bus Info Eπor Status and Misc Eπor Status registers, defined in the Polo Programmer's Guide, and controlled via the Bus Interface Reporting Register, also defined in the Polo Programmer's Guide. The eπor evente are:

1 : A bus error on a backplane bus

2: A voter eπor on a three way voted backplane signal

3: A thermal fault

4: A failure detected by the clock recovery chip

5: A board specific fault/failure. Note that unlike Stratebus systems, main memory single bit ECC errors do not generate maintenance interrupts on the first generation Polo CPU/ Memory board.

As described in section 9.4 on page 84, it is possible to turn off maintenance interrupts from certain sources so that a known hard failure does not flood the system with maintenance inteπupts. Specifically a voter error, a clock eπor, or a thermal fault may be disabled via the Bus

21 July 1995 l7

Xbus Functional Specification Stratus vmpany Confidential

Interface Fault Reporting register.

9.4.2.3 Software Control

Software may explicitly activate maintenance inteπupt by writing bit one of the Bus Interface Fault Reporting register.

9.4.2.4 Determining the Source of a Maintenance Interrupt

Upon seeing a maintenance interrupt in a system, the following process can be used to determine the source ot the inteπupt.

If a board that was in the system is no longer visible to software then that board generated a maintenance interrupt. To get more detail on what happened to the board, it can be reset and its Common Broken Status and ASIC Specific Broken Status registers examined. If multiple reset attempts have no effect, the board was either removed or is too broken to be read.

Boards that have not broken or not been removed from the system will have either the "Bus Interface Maint Int' bit or the "Board Logic Maint Int" bit set in the their Bus Interface Reporting register if they activated maintenance interrupt.

The "Board Logic Maint Int" is board specific and information on further identifying maintenance inteπupts with this bit set is discussed in individual board specs.

If the "Bus Interface Maint Int" bit is set, then Board Reset register may be read to see if a maintenance inteπupt was due to a board reset Similarly, the Bus tnfo Eπor Status and Misc Eπor stetus registers may be read to determine if the maintenance inteπupt was generated by any non-fatal eπors (the board would be broken otherwise). If no bits are set in either of these two registers, the maintenance interrupt was software generated.

9.5 Board Breaking Timing

Note in the figures below, lnfo_0's shows the time the info bus is driven to zero assuming that the bus is not already tristated from this side. xcever_disable shows the time the drivers from the broken board are disabled from the bus. Bus error is also shown when valid on the Xbus.

9.5.1 Board breaking and information latching.

There are several status bits in the Broken Status register on each board to determine the reason a board went broken. These bits are frozen after a broken condition occurs and will remain frozen until after a cold reset All bits in the common register are straight forward with the exception of board_speclf ic_set_broken. Board specific logic may have several reasons to assert this signal and may also have a similar register for the reasons it has asserted this signal. If board_speclf ic_broken is set in the common register, the board specific broken register will contain the real reason the board asserted the board_speclfic_set_broken signal. If a board breaks for a reason other than board specific broken, the board_specific_broken bit will not be set in the broken status register. However, board specific_set_broken may have been asserted after the original broken condition. This would have caused the board specific broken stetus register to latch and hold a reason it had set broken. It is important to understand that because the board_spectfic_broken bit is not set in the common broken status register, this was not the reason the board went broken. The reason the board went broken is logged in the broken status register and the board specific status register may be ignored.

| In the timing diagram below, someone_set >roken is the logical OR of the following signals:

21 July 1995 ~ ~~ 88

Xbus Functional .tus Company Confidential

cold_reset, warm_reset, slot_parity_erτor_set_broken, control_or_3way_vote_sig_erτor, asic_specific_set_broken, ecc_dbl_error_in. In addition Gambit contains the following signals: break jpcib, xb_parity gen_set_broken. The board will break if any broken signal is asserted on any phase (i.e. any 48mhz edge). my_setj roken_reg is the equivalent of otherside_set_broken_reg on the other ASIC. my_set_broken_reg is ORed with otherside_set_broken_reg to create set_broken_sync.

The common broken status register is frozen when latched_broken_status is true. Note, All stetus signals for the I/O register have the same timing as broken.

Note the bus eπor driven on the bus at the time the board went broken. This timing is important.

Figure 50. Board Breaking Timing and latching (arrived at phase_12_1)

Figure 51. Board Breaking Timing and latching, (arrived at phase_12_2)

21 July 1995 89

"

Xbus Functional Specification Stratus c mp ny Confidential

9.5.2 Board Breaking Timing on Info Loopback Error

If a loopback eπor occurs that will break the board, bus_error will be asserted in POST2 and the board will break in ERRL Both ASICs will know of the eπor because of there are four lines for each info bus that communicate loopback status between the ASICs (cmp_eπjn, cmp_eπ_out, drive_eπ_jn, drive_eπ_out).

Figure 52. Board Breaking Timing on Info Loopback Error

9.5.3 Board Breaking Timing on Heuristic or Arbitrary Broken.

Boards that break due to a Heuristic or an Arbitrary broken will participate in the entire eπor sequence and then deassert board_not_broken in ERR2.

Figure 53. Board Breaking Timing on Heuristic or Arbitrary Broken

OPERATION STATE ι POST2 I ERR1 ! CPUTEST |CPUP0STI IOTEST ' IOPOST1 i lOPOST2 I ERR2 INFO

24 MHZ

12_MHZ

BUS ERROR -L

BOARD_NOT_BROKt*4 INFO OE

CONTROL_.OE_

BUS_ERROR_FREEZi L

21 July 1995 90

"

Xbus Functional tus Company Confidential

10. Reset

In the Polo system, there are three kinds of reset. A cold reset is generated through a software command and implemented as a set of dedicated lines on the Xbus. a warm reset uses the same method. Finally, there is a power up/down reset These resets are described in this section.

10.1 Xbus Resets

Xbus resets are handled differently on Polo than they are on Golfbus systems. In Polo, an out of service bus cannot be used for transmitting information, like I/O writes to cause warm or cold resets. Instead, Polo implements a set of dedicated lines for this function. The operation of these lines is similar in protocol to the ReCC reset on Jette.

The three-way voted C/D-side comparison mechanism just described is such that if a board mistakenly drives a three-way voted signal due to a hardware failure, other boards will see that signal before the driving board breaks. For most of the control signals this does not adversely impact system integrity. An exception to this general rule are the warm and cold resets. Should a board mistakenly assert a reset line, it will eventually break itself, but not before potentially resetting one board or the entire system. This section describes how the Xbus addresses the above issues.

Drivers of reset are required to activate reset for 24MHz periods (1224MHz bus phases) in order to cause a warm reset, and 34MHz periods in order to cause a cold reset. An example of a warm reset is shown below.

Figure 54. Reset Signal Timing

1: At 1st 12_3, tha oppoalta alo* Cyclopβ ASIC check* the on-board reaat algnal

2: At tha next 48MHz dock (12_0), tha oppoalta aMa Cydopa drlvaa a aat broken algnal to tha bus ASIC

3: At tha next ttMHz dock (12_1), tha but ASIC reeetaea the aat broken algnal

4: Thlβ 12_1 dock edge la tha lataat the bua ASIC can receive the aet broken algnal and meet the llkiatrated tknkiglor deeaaartlng tha algnal on the bua.

5: At thla 13_3 edge, the bua traneoaiver drlvaa O's to all tha tranaoatvera so aa to ayπchronoualy αeaaaert the signals tftβ driving on tha bua.

6: Bua ASIC and external register disable transceivers here (latest reset will be disabled)

21 July 1995 91

?: O

Xbus Functional Specification Stratus company Confidential

10.2 Power System Generated Reset

Board reset from power related events, is generated by logic outside of the standard clocking of the AStCs. This is necessary due to the fact that ASIC DLLs have specific reset requirements that must be met There are three signals associated with the reset sequence. The power supply drives a signal called power_fault_. This signal is asserted when the power supply voltage is out of specification. ccJock_c_ and cc_lock_d_ are driven by the Sentry clock recovery chips when their clocks are out of specification. When all of these signals are de-asserted, logic high, the board level reset sequence is initiated. This consists of the phases described below. power_fautt_, cc_lock_c_, and cc_lock_d_ are ANDed together to form alert_. alert_ is used to release dumb logic, reset_n_c_, reset_n_d_, and trst_. The deassertion of reset_n(cd)_ causes the ASIC DLLs to begin their initialization sequence. This can take 409648Mhz clock ticks. At the end of this phase, good jwwer is asserted signalling that the DLLs should be settled. At that point, ttie bus ASICs can conclude the reset sequence and deassert broken.

On power down, as soon as power fault, ccJock_c_, or cc_lock_d_ is de-asserted, alert_ is asserted and the power down sequence begins. good_power is immediately de-asserted. The bus ASIC must immediately assert broken and reset 1 4Mhz dock later, trst_ and reset_n(c/d)_ is asserted and the power down sequence is complete. A state diagram for this sequence is shown in the figure below.

21 July 1995 92 t>

Xbus Functioni. ec»ι.«ation atus Company Confidential

Figure 55. Power Reset State Machine

lalert

alert

alert. = power__fault_ & cc_lock_c _ & cc_lock_d_

21 July 1995 93 vι

Xbus Functional Specification Stratus company Confidential

Figure 56. Board resets.

VCC @4.8v Asynchronous signal internal to power supply. Represents that the power supply is at 4.8 volts or greater.

POWER_FLT_ Asynchronous signal driven from power supply to reset logic only, represents that the power is within specification.

POWER_OK Synchronous signal derived from the alert_ and driven to only the bus ASIC.

CC_LOCK(C/D)_ Asynchronous signal from common clock chip driven to the reset logic. RESET_N(C/D)_ Asynchronous signal from the reset logic and driven to all ASICs with a DLL. This signal must be released before the ASIC's can lock their clocks. At this point it looks like this signal can be the equivalent of the power supply's DUMB signal.

OOL(C/D)_ Internal ASIC signal. Means the clock has achieved lock. COLD_RESET(C/D/)_ Synchronous signal driven from the bus ASIC and received by alt other ASICs. This signal initializes most internal logic.

WARM_RESET(C/D)_ Synchronous signal driven from the bus ASIC and received by all other ASICs. This signal initializes all internal logic.

TRST Asynchronous signal derived from DLL_RESET(C/D). Needed on all ASICs to initialize JTAG.

BROKEN Synchronous signal internal to the bus ASIC. Means the board is broken.

21 July 1995 94

Xbus Function.. e&..-ation itus Company Confidential

10.3 Fault Tolerant Issues:

POWER OK is a synchronous signal driven from asynchronous ALERT_. It is not possible to remove a single signal run here. However, If POWER_OK is stuck false, the board should never come out of cold reset and never clear broken. If POWER_OK is stuck true, broken will be guaranteed set by the OOL_ signal. When broken is set, it needs a reset complete signal to occur before the board will clear broken. This cannot occur because reset is not active.

The power supply design currentiy asserts ALERT when CCJ.OCK is lost. RESET_N_ should not be driven immediately after power or CC_LOCK_ is lost as this will cause even more clocks to go out of sync. RESET_N_ may be driven however after the board breaks.

An out of lock (OOL) condition on bus ASIC's will break the board but for other ASIC's we will rely on the out.ofjock condition to break ttie board for other C and D differences. Therefore a failure ot reset_n_c_ or reset_n_d_ to a bus ASIC is not a single point of failure. However, a failure of reset_n_(c/d)_ to non-bus ASIC's will not cause an immediate broken condition. On boards that do not have any other bus interfaces (CPU boards), the OOL condition will eventually break the board when bad data is transferred between ASIC's. If the back-end ASIC loses its lock on (I/O) boards that have buses in addition to the Xbus, the board may still compare data correctiy but not be able to talk to the bus in question due to timing violations. This should be taken under consideration on a board by board basis. If it is determined that broken should be set on an OOL condition then the boards may do so via the board_specific_set_broken signal.

10.4 ASIC Pins Required For Reset Functions.

The following pins are required for each ASIC. resetn_ The DM reset in signal trst_ JTAG reset. warm eset_ Warm reset out cold_reset_ Cold reset out

10.5 DUMB FETs

The Polo DUMB circuitry is a variation on the Stratabus boards' dumb/alert daughter card. This card provides dumb clamping when the board is in the power down and reset_count states.

21 July 1995 95

I

Xbus Functional Specification Stratus -ompany Confidential

11. Xbus Physical Partitioning

This section gives a high level overview of the physical partitioning for the interface ASICs to the Xbus. It describes the components necessary to interface to the Xbus as well as a functional description of how the interface is intended to operate. The fault tolerant strategy is also described.

11.1 Info Bus Partitioning

The C and D ASICs on the CPU board each drive half of the bits on the info bus when they are bus master. The C and D ASICs on the PCIB drive either half or all of the bits, depending on the type of transfer. During normal accesses, they drive half the bits as shown; during single-side accesses, they drive all of the bite, including the parity bit

Figure 57. Info Bus Partitioning

trid func_op Info Bus Parity

Bits arc assigned starting from the least significant bit.

Driven by the D-side ASIC Driven by the C-side ASIC during normal accesses, and during normal accesses, and by the driving ASIC during by the driving ASIC during single-side single-side accesses.

Parity is driven by the D-side on the CPU.

It is driven by the D-side of the PCIB during normal accesses, and by the driving ASIC during Single Side accesses.

11.2 Control Signal Partitioning

The D-side ASICs drive the control buses, and both C and D-side ASICs check.

11.3 Xbus Voted Signal Partitioning

The 3-way voted signals in a Polo system have varying topologies depending on function. In general, each ASIC on a board drives half of the 3-way voted signals originating on a board, and the other side ASIC sits at the end of the net and checks what is driven.

In the following diagrams, each transceiver section (shown as a NAND gate or inverter) of a 3 way voted triplet is in a separate package, to prevent a single failure from disturbing more than one bit ot the triplet.

21 July 1995 96

<

Xbus Functor Jpeo... cation ratus Company Confidential

Figure 58. board_not_broken_ routing

Figure 59. reset_ routing

21 July 1995 97

^

Xbus Functional Specification Stratus Confidential

Figure 60. sync_ routing

The sync signals are the only bidirectional 3way voted signals. Figure 61. online_ routing

he even_onlιne an o _onlιne signals are uni rec ona etween t e s.

11.3.1 ID PROM Partitioning

Golf/Jetta/Polo systems implement an IDPROM architecture based around a 2Kx8 EEPROM accessed via a JTAG scannable buffer. The Xbus permits ID PROM access under system software control as with the current Golfbus, but also supports:

1 ) IDPROM access as part of the board level boundary scan chain

2) IDPROM access on an un-powered freestanding board

These later two enhancements ease manufacturing's utilization of the IDPROM.

The IDPROM is JTAG 1149.1 accessible. JTAG is an industry standard 4 pin serial bus used for testing the interconnect between components on Polo boards. Initially the test bus will be driven from IBM compatible personal computers equipped with JTAG software/hardware accompanying all Polo debug stations.

A diagram of ttie ID PROM impiementetion is provided in figure 62 below. Normally, the illustrated jumper is always present The jumper block must be removed If manufactuπng wants to access the ID PROM without powering up the board (or before the board is tested or fully populated). Once the jumper is removed, the ID PROM can be accessed via the illustrated connector which provides both the JTAG signals and the power and ground to the ID PROM logic.

In normal operation (with the jumper present) the board comes out of the reset state with the IDPROM in the board's scan chain, i.e. the MUX is selecting the A inputs. Once software makes any access to the ID PROM logic through the bus ASICs ID PROM registers, the MUX B inputs are selected, thereby removing the ID PROM logic trom the board's normal scan chain and

21 July 1995 98 °

Xbus Function, .pecuication atus Company Confidential

connecting it to the bus ASIC instead. Figure 62. ID PROM Implementation

When MUX is in the "A" position, the IDPROM logic is part of the normal board scan chain. When MUX is in the "B" position, the IDPROM logic is removed from the board scan chain and is accessible from the bus ASICs.

Warm or Cold reset place MUX in the A position; any bus ASIC access places it in the B position until the next reset

21 July 1995 99

°\ o

1

Xbus Functional Specification Stratus ^mpany Confidential

12. Xbus Interface Block Diagrams

The Xbus interface portion of the ASICs that are on each board in the Polo system is referred to as the Xbus common logic. A single set of source files is maintained, and the logic is customized by Verilog ifdefs and included into the Gambit and Cyclops designs.

12.1 Xbus Top Level Block Diagram

12.1.1 Port List

The Xbus common logic connects to the ASIC specific logic within the bus ASICs, and to the Xbus side ASIC pins. Also included in the common logic (but not shown above) are miscellaneous support modules: the clock phasor generator, reset generator, DLL, active clock spine, etc.

Port lists for signals entering and leaving the Xbus common logic are discussed in the sections specific to each module.

12.1.2 Functional Description

The Xbus common logic provides all Xbus error checking and board stete management (broken, duplexed, on-line, etc.). It contains all control for IO register accesses, although the registers themselves are split between the common logic for bus related registers and the ASIC specific logic for board specific registers. However, the IO register control handshakes with and passes read data through the ASIC specific outbound control unit (OCU); this is because the OCU requirements vary greatiy between different board types and is best implemented as ASIC specific logic.

The inbound pipe provides a raw (before bus error and busy are available) version of the Xbus

21 July 1995 100

Xbus Function*, eciuυation itus Company Confidential

date to the IO registers, ASIC specific inbound pipe, and error logic for ack, busy, and error generation. Two (or more) phases later, it delivers qualified, known good data to the IO registers and ASIC specific inbound pipe for processing. On CPU boards, the inbound pipe also performs the IOVA address re-mapping.

The outbound pipe packages data for transmission on the Xbus. It does not buffer the data before assembly into a bus transaction; rather, it sends mux selects up into the ASIC specific logic as needed to construct the transaction. It stores a copy of the transaction in a recirculation pipe in case the data needs to be transmitted again as a result of a bus error.

12.2 Inbound Pipe Section

The inbound pipe watches the Xbus bus cycles and acknowledges, busies, or errors bus cycles based on information contained within the cycle. The inbound pipe also provides timed versions of the Xbus tor use internal to the ASIC. Figure 63 contains a block diagram of the logic contained within the inbound pipe.

21 July 1995 101

1 ^

9781

Xbus Functional Specification Stratus npany Confidential

Figure 63. Inbound Pipe Block Diagram ► indicates input output to block

JVC. X I ** A » iof o_a_reg, trid_a_reg, ftιnc_op_a_reg

** B * info_b_reg, trid_b_reg, func_op_b_reg

Info Postl Post2 PosU+1 phase phase phase phase

21 July 1995 102

Xbus Functiona. ecinoation atus Company Confidential

12.2.1 Port List

Table 19. Common Logic inbound Pipe Port List

21 July 1995 103

^

7/09781

Xbus Functional Specification Stratus mpany Confidential

Table 19. Common Logic Inbound Pipe Port List

21 July 1995 104

Xbus Functiona. .>eciι.wation itus Company Confidential

Table 19. Common Logic Inbound Pipe Port List

21 July 1995 105

Xbus Functional Specification Stratus company Confidential

Table 19. Common Logic inbound Pipe Port List

21 July 1995 106 ?

Xbus Functioni. peculation atus Company Confidential

Table 19. Common Logic Inbound Pipe Port List

21 July 1995 107 ea

Xbus Functional Specification Stratus ^-.mpany Confidential

Table 19. Common Logic Inbound Pipe Port List

12.2.2 Functional Description

The Xbus inbound pipe handles all issues relating to the inbound Xbus. This block generates bus error and the acknowledge (with some help from board specific logic). It also generates busy with input from four different sources: the I/O register block, the ping logic, the board specific logic, and the map RAM logic internal to this block.

The info and trid buses are registered and made available to the rest of the logic on the post_qual and pre.qual buses. The post and pre_qual trid, func_op, and info buses are simply referred to as the post qual or pre_qual bus in figure 63 in order increase readability. The I/O address map resides within this block so that IOVA Xbus addresses can be converted to system addresses before putting them on the pre-qual bus. A map error of any sort will inhibit the bus acknowledge and assert the signal map_error. The timing of the post and pre-qual buses relative to and Xbus cycle is shown in figure 64.

21 July 1995 108

Xbus Function., vpecwcation atus Company Confidential

Figure 64. Post-Qual and Pre-Qual Bus Timing

^■ info ^• postl DOSt2 info_a_reg ~X IOVA '.XPADOFFX prequal ^" X IOVA ^" X PAD postqual add valid

IOVA - I/O Virtual address PAD - Physical address

The valid signals indicate when the various buses can be used for address or data. Figure 65 illustrates the timing of data valid and lastjnfo for a 256 bit transfer on the Xbus. The timing for retum valid is identical to the timing for data_valid.

info_a_in ^D^^ Ε^ ^ y^^ into_a_reg

The page 2 SAM registers (1AM ASIC and Cyclops ASIC Map RAM Registers) are also located in this section within the Map RAM Control block (CPU only). These are the registers that control the I/O address table (i.e. Address Table Command, Address Table Data 2,1 , and 0 registers). These registers can be written directiy from the post.qual bus or read with the data being placed on the map_reg_out bus two bus phases after Post2 and kept there until the next read. It is also possible to read and write the map RAMs via these registers with that control also placed in the map RAM control block. Refer to section 12.62 on page 118 in the Polo Programming Guide for details on the registers internal to this block.

21 July 1995 109

1 O

Xbus Functional Specitt ition StratU j pany Confidential

12.3 PING Section

Figure 66. PING Control Block Diagram

The PING block controls generation of PINGs for I/O reads that are waiting for responses and acknowledgment of PINGs coming from other boards.

21 July 1995 110 -

Xbus Functiona. aciiicdtion tus Company Confidential

12.3.1 Port List

Table 20. PING Logic Port List

21 July 1995 111

9-»

T/US97/09781

Xbus Functional Specification Stratus .npany Confidential

Table 20. PING Logic Port List

21 July 1995 112

Xbus Functional ectøυdtjon tus Company Confidential

Table 20. PING Logic Port List

12.3.2 Functional Description

This module has two functional pieces: the send sub-module and the ack sub-module.

The send sub-module keeps track of the outbound read requests and their trids. This block also pings I/O read requests that have timed-out. The ping frequency is determined by the input read Ding interval from the common register block. During normal operation the interval will be 100 usec, however the ping interval can be set to 2.667 usec for testing purposes. The send module has an internal ping count used to keep track of when a cycle should be pinged.

The ack sub-module tracks incoming read requests and their status. It also acknowledges any incoming pings that hit a trid corresponding to an active read response being worked on. A trid table in the ping ack module keeps track of information regarding a read cycle that has been accepted by this ASIC. It maintains this information for acknowledging pings and for use during the read retum. Information specific to the read retum such as the trid and whether a given cycle is CD different is stored in the trid table. The table can hold up to seven entries: once it is full it sends out the signal ping_frfo_full_busy, indicating that the current transactions to be busied on the Xbus.

21 July 1995 113

?*><t

Xbus Functional Speciti αion Stratus mpany Confidential

12.4 Xbus Outbound Data and Control Unit

Figure 67. Outbound Data and Control Unit

XBUS OB Decode

XBUS UB Wave X4J XBUU kcho Slave (X4)

XBUS OB Master XBUS OB Peer-to-Peer Control Echo Master Control

(IO devices only)

21 July 1995 114

Xbus Functional .ac i ation tus Company Confidential

12.4.1 Port List

Table 21 describes the port list to the outbound data path.:

Table 21. Common Logic Outbound Pipe Port List

21 July 1995 115

^' o t*

Xbus Functional Specification Stratus company Confidential

Table 21. Common Logic Outbound Pipe Port List

21 July 1995 116

Xbus Functioi ipe .ication .ratus Company Confidential

Table 21. Common Logic Outbound Pipe Port List

21 July 1995 117 Ol

Xbus Functional Specification Stratus mpany Confidential

12.4.2 Functional Description

The Xbus outbound data and control unit issues all outbound transactions onto the Xbus including the echo of Peer-to-Peer transactions. The XBO will load all of the necessary information from the OCU upon the assertion of opjeady by the OCU. The XBO then issues the Xbus transaction onto the Xbus following the Xbus protocol. The XBO is comprised of the following functional blocks as illustrated in figure 67:

• XBO Decode - This block decodes the function supplied by the OCU.

INFO Registers - Five registers, each holding one info phase worth of data.

XBO Control - Five control sequencers, one master and four slaves. Each sequencer controls one info cycle of the transaction.

• ECHO Registers - Five registers, each holding one echo phase worth of data for Peer-to-Peer echo.

• XBO Echo Control - Five control sequencers, one master and four staves. Each sequencer controls one info cycle of the Peer-to-Peer echo transaction.

• Outbound Mux - This multiplexer channels the appropriate data onto the info jaut bus which is ultimately sent onto the Xbus.

• Swap Table - A four entry table, one entry for each PCI, which stores IOVA read byte swap information for outstanding read requests. This table is loaded following an IOVA Map RAM lookup for a read request The XBO uses this information when issuing the data retum.

Figure 68 Illustrates an example of XBO timing and its handshake with the OCU block. This particular illustration is that of a 128 bit write. Figure 69 and figure 70 illustrate the control flows tor the master and slave sequencers, respectively. The master sequencer controls the first info phase to be posted on the Xbus, each of the four slave sequencers control the other possible Xbus info

21 July 1995 118 ° ^<

Xbus Functional acih ution tus Company Confidential

cycles

Figure 71 illustrates the issuing of a peer-to-peer transaction onto the Xbus. Note that the issuing controller remains involved in the transaction until the transaction is echoed by its neighbor. The issuing controiler checks for error on the info cycles that it transmits but it does not check for busy since it is only valid on the echo. Therefore the issuing controller checks for busy on the echo portion of the transaction and retransmits the complete transaction in that occurrence.

Figure 72 illustrates the echo peer-to-peer timing. The XBO loads the post-qual info into its echo info registers when it receives a peer-to-peer indication in the post jual rid. It then transmits the received data onto the Xbus, checking for bus error and driving gmtjnh. In the case of an error during the echo, it retransmits the info. Figure 73 illustrates the echo master and slave control sequencers. Similar to the outbound sequencers, the master controls the first info phase and the slaves each control the other possible info phases of the echo.

21 July 1995 119

^" o

2 £ ^β » t i . ? m > x ϊ S, ! 5 5 5 5 δ, £ i 3 : ^§ j i o' -. s — o * E, & ^s

21 July 1995 120

Xbus Functional 3ci1.- -.Jon tus Company Confidential

Figure 69. Outbound Master Control

IDLE

MASTER CONTROL

21 July 1995 121 o il-

Xbus Functional Specification Stratus .npany Confidential

Figure 70. Outbound Slave Control

-op eng h +

SLAVE CONTROL

21 July 1995 122

Xbus Functiona 3Cifι„..don tus Company Confidential

Figure 71. Outbound Peer-to-Peer Timing

1 I H

Xbus Functional Specifiu.«ιon Stratus ipany Confidential

21 July 1995 124 S

Xbus Functional scih^tion tus Company Confidential

Figure 73. Outbound Peer-to-Peer Control

-p2p_valid + -last _j nfo

IMi ^" CONTROL NTROL

21 July 1995 125

3 l

12.5 Arbitration Logic

Figure 74. Arbitration Block Diagram

ARBITRATION BLOCK DIAGRAM

Xbus Functions eαt^ _^ αon tus Company Confidential

12.5.1 Port List

Table 22. Arbitration Functional Block Port List

12.5.2 Functional Description

Polo uses a distributed arbitration system. Each CPU and each PCIB has its own arbitration controller. Each board also has connections to the other three boards' control signals and is aware of the arbitration rules. This means each board can keep track of who is arbitrating and who should win. For more detail please refer to section 6.6 on page 40.

The main function of this logical block is to determine when the requesting node captures the Xbus. To do this it maintains a pπority address which consistent whh that of each of the other three nodes. The arbitration logic also determines when to block the node's request from being issued onto the Xbus. The arbitration control block is compnsed of the following functional blocks:

• PRIO_ADD_CNT - This block maintains the prioπty address for the resident node, this value is determined by the slotjd. The priority address is initialized to a value identical to that of its slotjd whenever a maintenance interrupt occurs.

ARB_PRIORITY - This block determines whether or not the resident node should be granted the XBUS based on the outstanding requests as well as the current priority address. ARB_CONTROL - This block maintains the necessary state information to delineate arbitration cycles. This block also initiates the updating of the pnoπty address in between

21 July 1995 127

Xbus Functional Specification Stratus npany Confidential

arbitration cycles. Figure 75. Arbitration State Machine

WφT* ATTON PONTROL

21 July 1995 128

3 IS

21 July 1995 129

S^

Xbus Functional Specifi _-ion Stratus npany Confidential

12.6 IO Register Section

| Figure 77. IO Register Block Diagram

21 July 1995 130 > 2 ;

Xbus Functiona eciti^ation .tus Company Confidential

12.6.1 Port List

Table 23. Common Logic Register Control Port List

21 July 1995 131

32-^

Xbus Functional Specification Stratus .mpany Confidential

Table 23. Common Logic Register Control Port List

21 July 1995 ^' ill 132

21 July 1995 133

3ZI,

21 July 1995 134 ^

Xbus Functiona eduction .tus Company Confidential

Table 23. Common Logic Register Control Port List

12.6.2 Functional Description

This block contains a sub-block which controls the reading and writing of all registers within the bus ASICs. In addition, it contains the a sub-block for the control of the ID-PROM registers, a sub- block containing several of the common registers, and a sub-block that is used to mux all of the register read sources together. The timing for a register write is shown in figure 78 and the timing for an Xbus register read is shown in figure 79

Figure 78. Register Write Timing

info postl post2 post3 post4 load_reg_xxx io_reg_write_data IX AUDK reg_xxx (VALID

Figure 79. Register Read Timing

Xbus Functional Specification Stratus, jmpany Confidential

21 July 1995 136 ι

Xbus Function, pecmoation atus Company Confidential

12.7.1 Port List

Table 24 describes the port list to the error checking and registering logic: Table 24. Error Checking and Registering Port List

21 July 1995 137

IΉ

Xbus Functional SpecitiMttion Stratuv <mpany Confidential

21 July 1995 138

- ≤m

Xbus Function pecmcation ratus Company Confidential

Table 24. Error Checking and Registering Port List

21 July 1995 139

S3-

Xbus Functional Specifi -on Stratus npany Confidential

Table 24. Error Checking and Registering Port List

21 July 1995 40

3i

Xbus Functioni peculation atus Company Confidential

Table 24. Error Checking and Registering Port List

21 July 1995 141

Xbus Functional Specituxition Stratui mpany Confidential

Table 24. Error Checking and Registering Port List

21 July 1995 142

Xbus Function jpecuication ratus Company Confidential

Table 24. Error Checking and Registering Port List

21 July 1995 143 izt,

Xbus Functional Specificuuon Stratus .ipany Confidential

Table 24. Error Checking and Registering Port List

21 July 1995 144

V

Xbus Functiona ecif.^(ion ltus Company Confidential

Table 24. Error Checking and Registering Port List

21 July 1995 145 3 £

Xbus Functional Specification Stratus pany Confidential

Table 24. Error Checking and Registering Port List

21 July 1995 146

Xbus Functional cifi ion us Company Confidential

Table 24. Error Checking and Registering Port List

21 July 1995 147 3-S

Xbus Functional Specifiu-jon Stratus npany Confidential

Table 24. Error Checking and Registering Port List

21 July 1995 148

" 613

Xbus Functional jcifi ition us Company Confidential

Table 24. Error Checking and Registering Port List

12.7.2 Functional Description

The Error Checking and Registering logic performs two basic functions. It controls most functions of error checking and it registers incoming and outgoing Xbus signals. The error checking function performs the following tasks:

ECC generation and checking/correcting

Parity generation and checking

Loopback checking

Three-way voting

Controls error protocol (includes decisions on breaking board)

Provides logic for error insertion

Records error information

The registering logic performs the following tasks:

• registers incoming bus signals

• registers outgoing bus signals

• combines (logical Or') neighbor, opposite, peer versions of registered ECC corrected control signals coming into the ASIC with the delayed outgoing corresponding signal to produce a combined control signal for internal ASIC module use

The Xbus can be broken into four classes of signals when discussing error checking:

Xbus Signals: these signals are protected by one parity bit this detects an odd number of errors. During normal accesses the D-side gate array drives approximately half of the signals and the C- side drives the other half. Please refer to Section 11.1 on page 96 for more detail on the signals that are driven by D-side and C-side. The D-side checks the data it has driven (loopback drive check) and the D-side checks the data that the C-side has driven with what the D-side would have driven (loopback compare check). During single sided accesses (PCIB only) the driving ASIC drives all of the bus signals. The driving ASIC can do a loopback drive check but the other ASIC cannot do a loopback compare check because it has no knowledge of the exact data being driven.

Control Bus Signals: these signals are protected by four bits of ECC, this detects a double bit error and can correct a single bit error. The D-side drives the signals and the C-side does the loopback compare check. The D-side does the loopback drive check.

Voted signals: these signals are triplicated and protected by three-way voting, this allow for one out of the three lines to be in error. The D-side drives the signals and the C-side does the loopback

21 July 1995 149

3<l

Xbus Functional Specify on Stratus npany Confidential

compare check. The D-side does the loopback drive check. The one exception to D-side driving all signals is the board iot Joroken_ signal. The C-side will drive the board iot >roken_ signal and the D-side will drive the enable for the board_not_broken_ signal.

Miscellaneous signals: The slot id signals are duplicated and if they are in disagreement the board will break.

Figure 81 shows the Bus Error flow chart, please refer to section 6.4 on page 30 for a detailed explanation of errors.

Figure 81. Bus Error Operation Flow Chart

21 July 1995 150

Xbus Functiona. ecii. ation atus Company Confidential

13. Xbus/Golfbus Differences

Table 25. XBusGolfbus Differences

21 July 1995 151 ^

Xbus Functional Speciti ation Stratus, mpany Confidential

Table 25. XBus/Golfbus Differences

21 July 1995 152 s^3