Linux/390

Common Device Support

The following section was copied from the Documentation/390 directory of the Linux distribution. It was written by Indo Adlung and is copyright IBM 1999, under the GNU Public License.

The next sections describe the functions, other than do_IRQ() in more details. The do_IRQ() interface is not described, as it is called from the Linux/390 first level interrupt handler only and does not comprise a device driver callable interface. Instead, the functional description of do_IO() also describes the input to the device specific interrupt handler.

General Information

Miscellaneous function prototypes, data declarations, and macro definitions can be found in the architecture specific "C header file" linux/arch/s390/kernel/IRQ.h.

Overview of CDS interface concepts

During its startup the Linux/390 system checks for peripheral devices. A so-called "subchannel" uniquely defines each of those devices by the ESA/390 channel subsystem. While the subchannel numbers are system generated, each subchannel also takes a user-defined attribute, the so-called "device number". Both, subchannel number and device number can not exceed 65535. The init_IRQ() routine gathers the information about control unit type and device types that imply specific I/O commands (channel command words or CCWs) in order to operate the device. Device drivers can retrieve this set of hardware information during their initialization step to recognize the devices they support using get_dev_info_by_IRQ() or get_dev_info_by_devno() respectively.

This methods implies that Linux/390 does not require to probe for free (not armed) interrupt request lines (IRQs) to drive its devices with. Where applicable, the device drivers can use the read_dev_chars() to retrieve device characteristics. This can be done without having to request device ownership previously.

When a device driver has recognized a device it wants to claim ownership for, it calls request_IRQ() with the device's subchannel id serving as pseudo IRQ line. One of the required parameters it has to specify is dev_id, defining a device status block, the CDS layer will use to notify the device driver's interrupt handler about interrupt information observed. It depends on the device driver to properly handle those interrupts.

In order to allow for easy I/O initiation the CDS layer provides a do_IO() interface that takes a device specific channel program (one or more CCWs) as input sets up the required architecture specific control blocks and initiates an I/O request on behalf of the device driver. The do_IO() routine allows for different I/O methods, synchronous and asynchronous, and allows to specify whether it expects the CDS layer to notify the device driver for every interrupt it observes, or with final status only. It also provides a scheme to allow for overlapped I/O processing. See "2.9 do_IO() - Initiate I/O Request" on page * for more details. A device driver must never issue ESA/390 I/O commands itself, but must use the Linux/390 CDS interfaces instead.

For long running I/O request to be canceled, the CDS layer provides the halt_IO() function. Some devices require to initially issue a HALT SUBCHANNEL (HSCH) command without having pending I/O requests. This function is also covered by halt_IO().

When done with a device, the device driver calls free_IRQ() to release its ownership for the device. During free_IRQ() processing the CDS layer also disables the device from presenting further interrupts: the device driver does not need to assure it. The device will be re-enabled for interrupts with the next call to request_IRQ().

get_dev_info_by_() - Retrieve Device Information

During system startup - init_IRQ() processing - the generic I/O device support checks for the devices available. For all devices found it collects the Sense-ID information. For those devices supporting the command it also obtains extended Sense-ID information.

int get_dev_info_by_IRQ( int IRQ, dev_info_t *devinfo);
int get_dev_info_by_devno( unsigned int IRQ, dev_info_t *devinfo);

typedef struct {
unsigned int devno; /* device number */
unsigned int status; /* device status */
senseid_t sid_data; /* senseID data */
} dev_info_t;

//
// SenseID response buffer layout
//
typedef struct {
/* common part */
unsigned char reserved; /* always 0x'FF' */
unsigned short cu_type; /* control unit type */
unsigned char cu_model; /* control unit model */
unsigned short dev_type; /* device type */
unsigned char dev_model; /* device model */
unsigned char unused; /* padding byte */
/* extended part */
ciw_t ciw[62]; /* variable # of CIWs */
} senseid_t;

typedef struct _ciw {
unsigned int et : 2; // entry type
unsigned int reserved : 2; // reserved
unsigned int ct : 4; // command type
unsigned int cmd : 8; // command
unsigned int count : 16; // count
} ciw_t;
Possible CIW entry types are:
#define CIW_TYPE_RDC 0x0; // read configuration data
#define CIW_TYPE_SII 0x1; // set interface identifier
#define CIW_TYPE_RNI 0x2; // read node identifier

The get_dev_info_by_IRQ() / get_dev_info_by_devno() functions return:

Usage Notes

In order to scan for known devices a device driver should scan all IRQs by calling get_dev_info() until it returns -ENODEV as there are not any more available devices.

If a device driver wants to request ownership for a specific device it must call request_IRQ() prior to be able to issue any I/O request for it, including above mentioned device dependent commands.

get_IRQ_by_devno() - Convert device identifiers

int get_IRQ_by_devno( unsigned int devno );
unsigned int get_devno_by_IRQ( int IRQ );

read_dev_chars() - Read Device Characteristics

In case of a registered interrupt handler, the interrupt handler must be able to properly react on interrupts related to the read_dev_chars() I/O commands. While the request is processed synchronously, the device interrupt handler is called for final ending status. In case of error situations the interrupt handler may recover appropriately. The device IRQ handler can recognize the corresponding interrupts by the interruption parameter being 0x00524443. If using the function with an existing device interrupt handler in place, the IRQ must be locked prior to call read_dev_chars().

int read_dev_chars( int IRQ, void **buffer, int length );

Usage Notes

1. If the caller does not provide a data buffer, read_dev_chars() allocates a data buffer and provides the device characteristics together. It is the caller's responsibility to release the kernel memory if not longer needed. This behavior is triggered by specifying a NULL buffer area (*buffer == NULL).
2. Alternatively, if the user specifies a buffer area himself, nothing is allocated.

request_IRQ() - Request Device Ownership

As previously discussed a device driver will scan for the devices its supports by calling get_dev_info(). Once it has found a device it will call request_IRQ() to request ownership for it. This call causes the subchannel to be enabled for interrupts if it was found operational.

int request_IRQ( unsigned int IRQ, int (*handler)( int, void *, struct pt_regs *),
 unsigned long irqflags, const char *devname, void *dev_id);

typedef struct {
unsigned int devno; /* device number from irb */
unsigned int intparm; /* interrupt parameter */
unsigned char cstat; /* channel status - accumulated */
unsigned char dstat; /* device status - accumulated */
unsigned char lpum; /* last path used mask from irb */
unsigned char unused; /* not used - reserved */
unsigned int flag; /* flag : see below */
unsigned long cpa; /* CCW addr from irb at prim. status */
unsigned int rescnt; /* count from irb at primary status */
unsigned int scnt; /* sense count, if available */
union {
irb_t irb; /* interruption response block */
sense_t sense; /* sense information */
} ii; /* interrupt information */
} devstat_t;

During request_IRQ() processing, the devstat_t layout does not matter as it won't be used during request_IRQ() processing. See "2.9 do_IO() - Initiate I/O Request" on page * for a functional description of its usage.

The request_IRQ() function returns :

Usage Notes

While Linux for Intel defines dev_id as a unique identifier for shared interrupt lines it has a totally different purpose on Linux/390. Here it serves as a shared interrupt status area between the generic device support layer, and the device specific driver. The value passed to request_IRQ() must therefore point to a valid devstat_t type buffer area the device driver must preserve for later usage. That is, it must not be released prior to a call to free_IRQ().

If request_IRQ() was called in enabled state, or if multiple CPUs are present, the device may present an interrupt to the specified handler prior to request_IRQ() return to the caller already. This includes the possibility of unsolicited interrupts or a pending interrupt status from an earlier solicited I/O request. The device driver must be able to handle this situation properly or the device may become non-operational.

free_IRQ() - Release Device Ownership

A device driver may call free_IRQ() to release ownership of a previously acquired device.

void free_IRQ( unsigned int IRQ, void *dev_id);

Usage Notes

Unfortunately free_IRQ() is defined not to return error codes. That is, if called with wrong parameters a device may still be operational although there is no device driver available to handle its interrupts. Further, during free_IRQ() processing we may possibly find pending interrupt conditions. As those need to be processed, we have to delay free_IRQ() returning until a clean device status is found by synchronously handling them.

The call to free_IRQ() will also cause the device (subchannel) be disabled for interrupts. The device driver must not release any data areas required for interrupt processing prior to free_IRQ() return to the caller as interrupts can occur prior to free_IRQ() returning. This is also true when called in disabled state if either multiple CPUs are presents or a pending interrupt status was found during free_IRQ() processing.

disable_IRQ() - Disable Interrupts for a given Device

int disable_IRQ( unsigned int IRQ );

The disable-IRQ() routine may return:

Usage Notes

As described previously the disable processing may require extensive processing. Therefore disabling and re-enabling the device using disable_IRQ() or enable_IRQ() should be avoided and is not suitable for high frequency operations.

void disable_IRQ( int IRQ);

enable_IRQ() - Enable Interrupts for a given Device

This function is used to enable a previously disabled device (subchannel). See "2.7 disable_IRQ() - Disable Interrupts for a given Device" on page * for more details.

int enable_IRQ( unsigned int IRQ );

The enable-IRQ() routine may return:

do_IO() - Initiate I/O Request

The do_IO() routines is the I/O request front-end processor. All device driver I/O requests must be issued using this routine. A device driver must not issue ESA/390 I/O commands itself. Instead the do_IO() routine provides all interfaces required to drive arbitrary devices.

This description also covers the status information passed to the device driver's interrupt handler as this is related to the rules (flags) defined with the associated I/O request when calling do_IO().

int do_IO( int IRQ, ccw1_t *cpa, unsigned long intparm, unsigned int lpm, unsigned long flag);

typedef struct {
char cmd_code; /* command code */
char flags; /* flags, like IDA addressing, etc. */
unsigned short count; /* byte count */
void *cda; /* data address */
} ccw1_t __attribute__ ((aligned(8)));

The do_IO() function returns:

When the I/O request completes, the CDS first level interrupt handler will setup the dev_id buffer of type devstat_t defined during request_IRQ() processing. See "2.5 request_IRQ() - Request Device Ownership" on page * for the devstat_t data layout. The dev_id->intparm field in the device status area will contain the value the device driver has associated with a particular I/O request. If a pending device status was recognized dev_id->intparm will be set to 0 (zero). This may happen during I/O initiation or delayed by an alert status notification.

In any case this status is not related to the current (last) I/O request. In case of a delayed status notification no special interrupt will be presented to indicate I/O completion as the I/O request was never started, even though do_IO() returned with successful completion.

Possible dev_id->flag values are:

If device status DEVSTAT_FLAG_SENSE_AVAIL is indicated in field dev_id->flag, field dev_id->scnt describes the number of device specific sense bytes available in the sense area dev_id->ii.sense. No device sensing by the device driver itself is required.

typedef struct {
unsigned char res[32]; /* reserved */
unsigned char data[32]; /* sense data */
} sense_t;

The devi_id->cstat field provides the (accumulated) subchannel status:

In rare error situations the device driver may require access to the original hardware interrupt data beyond the scope of previously mentioned information. For those situations the Linux/390 common device support provides the interrupt response block (IRB) as part of the device status block in dev_id->ii.irb.

Usage Notes

Prior to call do_IO() the device driver must assure disabled state, that is, the I/O mask value in the PSW must be disabled. This can be accomplished by calling __save_flags(flags). The current PSW flags are preserved and can be restored by __restore_flags(flags) at a later time.

If the device driver violates this rule while running in a uni-processor environment an interrupt might be presented prior to the do_IO() routine returning to the device driver main path. In this case we will end in a deadlock situation, as the interrupt handler will try to obtain the IRQ lock the device driver still owns.

The device driver is allowed to issue the next do_IO() call from within its interrupt handler already. It is not required to schedule a bottom-half, unless an non deterministically long running error recovery procedure or similar needs to be scheduled. During I/O processing the Linux/390 generic I/O device driver support has already obtained the IRQ lock, that is, the handler must not try to obtain it again when calling do_IO() or we end in a deadlock situation. Anyway, the device driver's interrupt handler must only call do_IO() if the handler itself can be entered recursively if do_IO(), for example, it finds a status pending and needs to all the interrupt handler itself.

Device drivers should not rely on DOIO_WAIT_FOR_INTERRUPT synchronous I/O request processing too heavily. All I/O devices, but the console device are driven using a single shared interrupt subclass (ISC). For synchronous processing the device is temporarily mapped to a special ISC while the calling CPU waits for I/O completion. As this special ISC is gated, all synchronous requests in an SMP environment are serialized which may cause other CPUs to spin. This service is primarily meant to be used during device driver initialization for ease of device setup.

The lpm input parameter might be used for multi-path devices shared among multiple systems as the Linux/390 CDS is not grouping channel paths. Therefore, its use might be required if multiple access paths to a device are available and the device was reserved by means of a reserve device command (for devices supporting this technique). When issuing this command the device driver needs to extract the dev_id->lpum value and restrict all subsequent channel programs to this channel path until the device is released by a device release command. Otherwise a deadlock may occur.

If a device driver relies on an I/O request to be completed prior to start the next it can reduce I/O processing overhead by chaining a no-op I/O command CCW_CMD_NOOP to the end of the submitted CCW chain. This will force Channel-End and Device-End status to be presented together, with a single interrupt.

In order to minimize I/O overhead, a device driver should use the DOIO_REPORT_ALL only if the device can report intermediate interrupt information prior to device-end the device driver urgently relies on. In this case all I/O interruptions are presented to the device driver until final status is recognized.

If a device is able to recover from asynchronously presented I/O errors, it can perform overlapping I/O using the DOIO_EARLY_NOTIFICATION flag. While some devices always report channel-end and device-end together, with a single interrupt, others present primary status (channel-end) when the channel is ready for the next I/O request and secondary status (device-end) when the data transmission has been completed at the device.

Unless the channel subsystem at any time presents a secondary status interrupt, exploiting this feature will cause only primary status interrupts to be presented to the device driver while overlapping I/O is performed. When a secondary status without error (alert status) is presented, this indicates successful completion for all overlapping do_IO() requests that have been issued since the last secondary (final) status.

During interrupt processing the device specific interrupt handler should avoid basing its processing decisions on the interruption response block (IRB) that is part of the dev_id buffer area. The IRB area represents the interruption parameters from the last interrupt received. Unless the device driver has specified DOIO_REPORT_ALL or is called with a pending status (DEVSTAT_STATUS_PENDING), the IRB information may or may not show the complete interruption status, but the last interrupt only. Therefore the device driver should usually base its processing decisions on the values of dev_id->cstat and dev_id->dstat that represent the accumulated subchannel and device status information gathered since do_IO() request initiation.

halt_IO() - Halt I/O Request Processing

Sometimes a device driver might need a possibility to stop the processing of a long-running channel program or the device might require to initially issue a halt subchannel (HSCH) I/O command. For those purposes the halt_IO() command is provided.

int halt_IO( int IRQ, /* subchannel number */
 int intparm, /* dummy intparm */
 unsigned int flag); /* operation mode */

The halt_IO() function returns:

Usage Notes

A device driver may write a never-ending channel program by writing a channel program that at its end loops back to its beginning by means of a transfer in channel (TIC) command (CCW_CMD_TIC). Usually network device drivers perform this by setting the PCI CCW flag (CCW_FLAG_PCI). Once this CCW is executed a program controlled interrupt (PCI) is generated. The device driver can then perform an appropriate action. Prior to interrupt of an outstanding read to a network device (with or without PCI flag) a halt_IO() is required to end the pending operation.

We do not allow the stopping of synchronous I/O requests by means of a halt_IO() call. The function will return -EBUSY instead.

Miscellaneous Support Routines

These two macro definitions are required to obtain the device specific IRQ lock. The lock needs to be obtained if the device driver intends to call do_IO() or halt_IO() from anywhere but the device interrupt handler (where the lock is already owned). Those routines must only be used if running disabled for interrupts already. Otherwise use s390irq_spin_lock_irqsave() and the corresponding unlock routine instead.

s390irq_spin_lock( int IRQ);
s390irq_spin_unlock( int IRQ);

These two macro definitions are required to obtain the device specific IRQ lock. The lock needs to be obtained if the device driver intends to call do_IO() or halt_IO() from anywhere but the device interrupt handler (where the lock is already owned). Those routines should only be used if running enabled for interrupts. If running disabled already, the driver should use s390irq_spin_lock() and the corresponding unlock routine instead.

s390irq_spin_lock_irqsave( int IRQ, unsigned long flags);
s390irq_spin_unlock_irqrestore( int IRQ, unsigned long flags);

Special Console Interface Routines

This section describes the special interface routines required for system console processing. Though they are an extension to the Linux/390 device driver interface concept, they base on the same principles. It was necessary to build those extensions to assure a deterministic behavior in critical situations, for example, printk() messages by other device drivers running disabled for interrupts during I/O interrupt handling or in case of a panic() message being raised.

set_cons_dev() - Set Console Device

This routine allows specification of the system console device. This is necessary as the console is not driven by the same ESA/390 interrupt subclass as are other devices, but it is assigned its own interrupt subclass. Only one device can act as system console. See wait_cons_dev() for details.

int set_cons_dev( int IRQ);

The set_cons_dev() function returns

reset_cons_dev() - Reset Console Device

This routine allows for resetting the console device specification. See "2.12.1 set_cons_dev() - Set Console Device" on page * for details.

int reset_cons_dev( int IRQ);

The reset_cons_dev() function returns

wait_cons_dev() - Synchronously Wait for Console Processing

The wait_cons_dev() routine is used by the console device driver when its buffer pool for intermediate request queuing is exhausted and a new output request is received. In this case the console driver uses the wait_cons_dev() routine to synchronously wait until enough buffer space is gained to enqueue the current request. Any pending interrupt condition for the console device found during wait_cons_dev() processing causes its interrupt handler to be called.

int wait_cons_dev( int IRQ); 

The wait_cons_dev() function returns :

Usage Notes

The function should be used carefully. Especially in a SMP environment the wait_cons_dev() processing requires that all but the special console ISC are disabled. In a SMP system this requires the other CPUs to be signaled to disable/enable those ISCs.

Major and Minor Numbers

Major Block Device 95 - IBM S/390 DASD storage

0 = /dev/dasd0 	First DASD device, major
1 = /dev/dasd0a First DASD device, block 1
2 = /dev/dasd0b First DASD device, block 2
3 = /dev/dasd0c First DASD device, block 3
4 = /dev/dasd1 	Second DASD device, major
5 = /dev/dasd1a Second DASD device, block 1
6 = /dev/dasd1b Second DASD device, block 2
7 = /dev/dasd1c Second DASD device, block 3

Major Block Device 96 - IBM S/390 VM/ESA minidisk

0 = /dev/mnd0 	First VM/ESA minidisk
1 = /dev/mnd1 	Second VM/ESA minidisk

DASD Device Driver

The following section was copied from the Documentation/390 directory of the Linux distribution. It was written by Indo Adlung and is copyright IBM 1999, under the GNU Public License.

Usage

Low-level format

For using an ECKD-DASD as a Linux hard disk you have to low-level format the tracks by issuing the BLKDASDFORMAT-ioctl on that device. This will erase any data on that volume including IBM volume labels, VTOCs etceteras. The ioctl may take a 'struct format_data *' or 'NULL' as an argument.

typedef struct {
	int start_unit;
	int stop_unit;
	int blksize;
} format_data_t;

When a NULL argument is passed to the BLKDASDFORMAT ioctl the whole disk is formatted to a blocksize of 1024 bytes. Otherwise start_unit and stop_unit are the first and last track to be formatted. If stop_unit is -1 it implies that the DASD is formatted from start_unit up to the last track. blksize can be any power of two between 512 and 4096. We recommend no blksize lower than 1024 because the ext2fs uses 1kB blocks anyway and you gain approximately 50% of capacity increasing your blksize from 512 byte to 1kB.

Make a filesystem

Then you can mk??fs the filesystem of your choice on that volume or partition. For reasons of sanity you should build your filesystem on the partition /dev/dd?1 instead of the whole volume. You only lose 3kB but may be sure that you can reuse your data after introduction of a real partition table.

Bugs

• Performance sometimes is rather low because we do not fully exploit clustering

TODO-List

• Add IBM'S Disk layout to genhd
• Enhance driver to use more than one major number
• Enable usage as a module
• Support Cache fast write and DASD fast write (ECKD)

Files added to the Linux Distribution

Linux System Calls

Control Register Usage

Control Register 0

Control Register 1

Control Registers 2-5

Control Register 6

Control Register 7

Control Register 8

Control Registers 9-11

Control Register 12

Control Register 13

Control Register 14

Control Register 15

Access Register Usage

IPL under VM/ESA

Initial RAMDISK

Root Filesystem on VM Minidisk