Title of Invention

A METHOD FOR OPERATING A SERVER FARM

Abstract

The invention related to a method for operating a server farm having an arrangement of a plurality of computers (1,2,3,4) which are set up to execute a multiplicity of software units (A1..A4, D1...D5) and an additional monitoring computer (5) for monitoring the computers (1,2,3,4) and for intervening in the computers (1,2,3,4), and the status of the computers (1,2,3,4) being repeatedly monitored. The software units (A1..A4, D1..D5) are assigned a weightage (L2K,SL) in accordance with their importance, and the following method steps are carried out in the event of a computer (1,2,3,4) failure: information regarding which of the computers (1,2,3,4) have failed at the monitoring time and which software units (A1..A4, D1..D5) are affected thereby is collected, a multiplicity of, i.e. at least two, possible transfer scenarios for transferring the software units (A1..A4, D1..D5) which are affected by the failure or failures to other computers (1,2,3,4) is determined, a transfer scenario respectively indicating which software units (A1..A4, D1..D5) are transferred to which computers (1,2,3,4), the transfer scenarios are assessed using the weightage (L2K,SL) of the software units (A1..A4, D1..D5), a transfer scenario is selected on the basis of the assessment, and the selected transfer scenario is carried out. {FIG. 4}

Full Text

Description Method for operating an arrangement of a plurality of computers in the event of a computer failure The invention relates to a method for operating an arrangement of a plurality of computers which are set up to execute software units, provision being made of an additional monitoring computer for monitoring the computers and for intervening in the computers, and the status of the computers being repeatedly monitored. Computer arrangements of this type are known, inter alia, under the term server farm. Server farms usually comprise a large number of identical servers which are also referred to below as computers and on which the same or different services or applications run. Since all servers execute the services and applications independently of one another, faults in one server do not have a direct influence on the rest of the server farm. User inquiries are distributed to all members of the server farm in accordance with defined rules. This is the task of the monitoring computer. Mechanisms which have been implemented for distributing the load ensure that use of the individual servers corresponds to the respective processing capacity. The failure of one server is relatively unproblematic since only a few services and applications are affected thereby. Since the individual computers are relatively small and thus inexpensive devices, it is unproblematic, from the point of view of costs, to keep one or more standby computers ready, to which, after a computer has failed, the software units affected, i.e. services and applications, for example, are transferred in order to thus restore normal operation. However, server farms often comprise several hundred computers . In the case of these so-called blade servers, there is no need for the external wiring complexity since the computers are accommodated and connected in the form of a plug-in card. However, the problem with this design is that, in the event of a power supply unit, for example, failing, a plurality of computers are affected and thus a plurality of computers simultaneously fail. For economic reasons, it is not possible to provide, for every case, as many standby computers as are actually required on account of the failure. Satisfactory operation of the computer arrangement is thus not ensured in every case. When a software unit fails on account of the failure of a computer, it is known practice to transfer the software unit to a standby computer, that is to say to restart it there. If a plurality of computers fail, a plurality of software units are therefore affected. An attempt is then made, for each software unit, to find a standby computer or a sufficiently large amount of free capacity in a standby computer in order to be able to restart the software unit. This results in competitive situations, thus jeopardizing fault-free further operation. It is an object of the invention to specify a method for operating an arrangement of a plurality of computers, in which the best possible availability of the software units which are executed in the computer arrangement is ensured. In this case, both the failure of an individual computer and the failure of a plurality of computers are intended to be taken into account. This object is achieved by means of a method of the type mentioned initially which is characterized in that the software units are assigned a weighting in accordance with their importance, and the following- method steps are carried out in the event of a computer failing: information regarding all of the computers which have failed at the monitoring time and software units affected thereby is collected, and a transfer scenario for transferring the software units which are affected by the to other computers is determined using the weighting of the software units. The method according to the invention follows the finding that, under certain circumstances, a computer having sufficient free capacity cannot be found for all software units. A transfer scenario which specifies which software units are transferred to which computers is sought. Weighting the software units in accordance with their importance achieves the advantage that a transfer scenario which, when considered overall, constitutes an optimal solution is determined, the more important software units thus preferably being transferred to another computer, while unimportant software units are not restarted. The decisive advantage of the present method according to the invention is that the transfer of software units to other computers is determined not only with regard to an individual software unit but rather the server farm with its software units is considered as a whole in order to disrupt overall operation as little as possible. It is also advantageous that not only the failure of an individual computer can be taken into account but rather a plurality of computers which have simultaneously failed and the software units which have correspondingly failed are also included in the determination of a transfer scenario. In one advantageous refinement of the method according to the invention, a multiplicity of possible transfer scenarios are first of all determined and these scenarios are then assessed using the weighting of the software units. The assessment of the overall scenario can be used to discern which is the optimal scenario overall. In one development of the invention, the process of determining a transfer scenario includes the fact that active software units are terminated in other computers in order to create free capacity for transferring software units from computers which have failed. In this case, it is advantageous that unimportant active software units are terminated in order to create capacity for transferring software units which are of greater importance and were previously active in one of the failed computers. In one favorable refinement, the weighting is effected using a plurality of attributes, a first "license to kill" attribute specifying a measure of the power to have other active software units terminated in order to create free capacity and a second "shutdown limit" attribute specifying a measure of the resistance to the request for termination by other software packages in order to be transferred from a computer which has failed. The first attribute is used to determine which of the failed software units are preferably transferred to a standby computer, that is to say are restarted there. The second attribute is used to find out which of the active software packages are terminated in order to create space for transferring software packages which were previously active on a computer which has now failed. In this case, it is favorable that the behavior of the computer arrangement, when transferring failed software units, can be influenced by an administrator by appropriately allocating attributes. Within the scope of this application, software units may be services, applications or packages from an operating system and services and/or applications. The packages which were mentioned last and which are also referred to as "images" are used, in particular, in large server farms having blade servers. When booting a server within the server farm, an image is loaded from a central memory and is executed in the server. If a second server is intended to be started using the same application, a copy of the image is simply loaded into the second server and executed there. In this way, the configuration complexity is minimal. Further advantageous refinements of the invention are specified in subclaims. The invention will be explained in more detail below using an exemplary embodiment. In the drawings: figure 1 shows an arrangement having a plurality of computers and a monitoring computer, to which the method according to the invention can be applied, figure 2 shows a table corresponding to a first transfer scenario, figure 3 shows a table corresponding to a second transfer scenario, and figure 4 shows a flowchart of one exemplary embodiment of the method according to the invention. Figure 1 illustrates an arrangement having a plurality of computers 1, 2, 3, 4 and a monitoring computer 5. The computers 1, 2, 3 and 4 do not have a connection which has been set up to interchange data. However, they are connected to the monitoring computer 5 via lines 6. These connections which have been implemented in any desired manner and use the lines 6 are used to monitor the computers 1, 2, 3 and 4 and to intervene, if necessary, in the sequences in these computers. In the example shown, an operating system 0S1 and an application A1 are active in the computer 1. Active means that the operating system and the application are in a normal operating state. An operating system 0S2 and an application A2 are active in the computer 2. In addition, two services D1 and D2 are running on the computer 2. An operating system 0S3 and two applications A3 and A4 as well as three services D3, D4 and D5 are active in the computer 3. The computer 4 is a standby computer in which neither an operating system nor an application nor a service is active in the operating state shown. The monitoring computer 5 monitors the status of the computers 1, 2 and 3 at regular intervals of time T in order to determine whether they are in a normal operating state or whether there is a fault state (see figure 2). The computer 4 does not need to be monitored since the monitoring computer 5 is aware of its status as a standby computer. Monitoring may be configured in such a manner that the monitoring computer 5 sends an inquiry to the computers 1, 2, 3 and 4 and then waits to see whether a response is sent back. However, detectors which' automatically monitor the operating state and transmit a fault message to the monitoring computer 5 if there is a deviation from the normal operating state may also be provided in the computers 1, 2, 3 and 4. Both types of monitoring may also be combined, that is to say the computers 1, 2, 3 and 4 automatically transmit fault reports but also respond to inquiries from the monitoring computer 5. If the description of the invention refers to the reception or transmission of inquiries or reports by the "computers", this includes both information processing using hardware and processing using software by the respective active operating systems, applications or services. The computers 1, 2 and 3 may be operated using different operating systems. It must merely be ensured that message or command transmission via the lines 6 functions in accordance with the same standard so that the monitoring computer 5 can communicate with all of the computers 1, 2, 3 and 4. In the embodiment described here, the computers 1, 2 and 3 are checked (step 11) at regular intervals of time T, as shown in figure 4. If a fault is not established in step 12, the process waits for the point in time at which a new check is intended to be carried out. If a fault was detected, information regarding the software units affected is then collected (step 14). In an example scenario, it is assumed that the computers 1 and 3 have failed. This fault was detected by the monitoring computer 5. In step 14, information regarding the software units which have been affected by the failures is then collected. In this case, it is established that the applications A1, A3 and A4 as well as the services D3, D4 and D5 have been affected by the failure. In addition, the available capacity is determined in a step 15 which subsequently takes place or takes place in parallel. In the present case, the computer 4 is available as a standby computer. If the capacity available in the latter does not suffice, it would be necessary to check the importance of the applications A2 and the services D1 and D2 which are active in the computer 2 in comparison with the importance of the applications A1, A3 and A4 and the services D3, D4 and D5 which are to be transferred. After the software units to be transferred and the available capacity have then been determined, possible scenarios for transferring the software units, which have been affected by the failure or failures, to other computers may be carried out (step 16) . A scenario which is optimal overall must then be selected from the multiplicity of possible scenarios. In this case, optimal overall means that both the interest in transferring failed software units to other computers is taken into account and operation of the software units, which are still active, is retained as completely as possible. Two situations occur, in principle. In the first situation, enough free capacity is available to be able to transfer all of the failed software units. In the other situation, the number of failed computers is greater than the number of standby computers. Therefore, in the submethods for determining the transfer scenario, it is first of all necessary to determine which of the failed software units are transferred to the standby computers. The importance of the remaining failed software units is investigated and it is determined whether or not they are transferred to another computer, the software units which are currently running on this computer being forced to be terminated. If it is not possible to find a new computer for all of the failed software units, the least important software units of the failed software units are determined and the entire process of determining the transfer scenario is started from the beginning. When creating possible transfer scenarios, it is also necessary to take into account whether a software unit is understood as meaning an application or a service or whether a software unit corresponds to an image. In the first case, the application or the services can be transferred to a computer in which other applications or services are already active and which is operated using a suitable operating system. In the second case, a completely free computer must be available since the image also comprises the operating system and only one image per computer is possible. One possible scenario for transferring applications and services is illustrated in the table shown in figure 2. Two attributes "license to kill" (L2K) and "shutdown limit" (SL) are assigned to the applications A1..A4 and to the services D1..D5. The first attribute L2K is a measure of the power to have other active software units terminated by the monitoring computer 5 in order to create free capacity. The second- attribute SL is a measure of the resistance to the request for termination by other software packages in order to be transferred from a computer which has failed. The fact of whether an application is terminated and sufficient capacity is thus provided for transferring another application thus depends on the "license to kill" attribute of the application to be transferred and on the "shutdown limit" attribute of the application which is still active and may possibly be terminated. The same applies in the case of a service. The attributes described are likewise assigned to said services. Returning to the scenario described in figure 1, according to which the computers 1 and 3 have failed, the computer 4 is available as a standby computer and an application A2 and two services D1 and D2 are already active in the computer 2, the optimal scenario shown in the table in figure 2 could result. The values entered in the table are to be understood such that 1 is the lowest value and 5 is the highest value. The higher the value, the higher the importance of the respective application or of the respective service. According to this example scenario, the applications A4 and A1 are restarted in the standby computer 4, that is to say are assumed by the latter. There is no longer sufficient free capacity in the computer 4 for the application A3. However, the attribute L2K has the value 4 which is greater than the value of the attribute SL of the application A2 . Comparing these values reveals that the application A2 is to be terminated so that the application A3 can be restarted in the computer 2. The services D1 and D2 can continue to run on the computer 2. The service D3 in the computer 3 is restarted in the computer 2. The services D4 and D5 no longer have any space in the computers 2 and 4 . They have a very low value as regards the "license to kill" attribute, with the result that no services or application is/are terminated in order to create free capacity for the services D4 and D5. These services therefore cannot be transferred to another computer. In figure 3, transfer for each application or service is not considered but rather, in the case of a transfer, only the entire image can be transferred. The image I1 comprises an operating system 0S1 and an application A1, as were active in the computer A1 before the failure. The image I2 comprises the operating system 0S2 which is running on the computer 2, the application A2 and the services D1 and D2. The image I3 comprises the operating system 0S3 which was active in the computer 3 before the failure, the applications A3 and A4 and the services D4 and D5. The images are in turn assigned the attributes L2K and SL. According to the example scenario shown in figure 3, the image I1 is restarted in the computer 4, the image 12 remains active in the computer 2 and no free computer can be found for the image 13. A1though the image 12 could be terminated in order to restart the image 13 in the computer 2, the "license to kill" attribute of the image I3 with the value 2 is less than the value of the "shutdown limit" attribute of the image I2. Therefore, the image I3 does not have the power to have the image I2 terminated in order to create free capacity. In step 17, the scenarios are thus assessed by linking the attributes of all active applications and all software units which are affected by the failure with the possible scenarios. In step 18, an optimal scenario is selected, on the basis of the weighting using the attributes in the situation described. In a step 19, the optimal transfer scenario determined is implemented, software units are terminated and failed software units are restarted in the newly determined computers, with the result that a transfer is effected. After this stable state has been reached, the method begins from the beginning with checking the status of the computers. The boundary conditions for creating a scenario follow defined rules. Rules which may play a role in forming transfer scenarios are described below: an image may simultaneously run only on one server and a server may simultaneously receive only one image; standby computers which are either idle or have been switched off are primarily used to transfer failed software units; if sufficient standby computers are not available, active computers are run down in order to receive images depending on the value of their attributes applications or services are correspondingly terminated in order to transfer failed services or applications; the submethods for determining the transfer scenario operate in such a manner that the selected transfer scenario has the most minor effects on the active computers - the selected transfer scenario should simultaneously satisfy the condition of transferring as many failed software units as possible; all important software units are first of all transferred to other computers in accordance with the abovementioned rules and the attributes which have likewise been described - if no transfer scenario which satisfies all of the requirements is possible, failed software units are excluded from the determination of a transfer scenario; there is only' one transfer per software unit software units which have been terminated cannot, for their part, terminate other active software units; there is no domino effect for computers having a low priority because they are frequently the victim of transfers. So that a transfer scenario can still be calculated, the status of the computers at a defined point in time is considered. This is repeated at defined intervals of time. A1l of the computer failures are handled at the same time in a single determination entity, so that the number of failed computers is fixed at a particular point in time and does not change while determining the transfer scenario. If further computers fail while calculating the transfer scenario, these failures are not taken into account until the next monitoring time. In an extended method, the ownership of the computers and the hardware of the computers can be taken into account in order to avoid "wasting" a powerful computer for a software unit which does not have sufficient importance for such a computer. Taking into account the ownership of a computer is important because it is customary in server farms for a customer to have its own server operated in the server farm. This inevitably results in such a computer not being available as a computer for transferring extraneous software units but rather only being able to be used by this customer. The ownership of the computers is taken into account using a third attribute "server group". A transfer scenario is only determined using computers which belong to the same "server group". The type of hardware of the computer is taken into account using a "server class" attribute. The more the failed computers correspond to the hardware of the computers which are taken into account for the transfer, the greater the affinity to this computer. "Suitable" is defined in consideration of the integer and string comparisons, as explained below. In one practical implementation, the attributes are stored in the form of strings or integers. If the individual attributes are compared, the operators ' , >=, ==, /=' are used. Only the operator ' ==' is permissible for strings. On the basis of these settings, all of the computers and software units can be compared using the operators which were specified above. The optimal transfer scenario can thus be determined. In this manner, all of the computers and all of the software units are investigated in order to determine whether they are a potential destination for a transfer. This is, of course, carried out only when it is necessary, that is to say when computers have failed and are to be transferred, if possible, to another computer. The main task is to find the correct computers or software units which are run down in sight of the computers or software units which are running. The determination of this scenario may be a lengthy process. In the case of a large server farm having several hundred computers S and no standby computers, the number of variations to be taken into account when determining the transfer scenario in the case of a small number of failed computers F is calculated as For a server rack having 20 computers, the number of permutations when five computers fail is greater than 3000. If a second server rack having a further 20 computers is added, the number of permutations is greater than 50,000. In order to find an optimal transfer scenario from this large number of possible transfer scenarios, it may be necessary to use special computation methods. It must be ensured that the overall influence of each transfer scenario is taken into account. The effects depend on the attributes which were mentioned above. While the possible transfer scenarios are being calculated, the effects on the overall system are advantageously continuously calculated and predicted, even at the same time. Only the scenarios in which it is possible to have more minor effects on the overall system than the transfer scenarios which have already been investigated are determined in detail. This requires a type of sorting of ' the permutations to be investigated in order to ensure that the solutions with the most minor effects are handled first. This significantly speeds up the determination process. The determination process is terminated in advance if a transfer scenario which does not have any effects on active software units has been found. A so-called greedy algorithm is preferably used during the calculation. The concept described is capable of finding a transfer scenario which influences the computers and software units (which are running) to as slight an extent as possible, as many software units as possible nevertheless being transferred. The method according to the invention can also be used in virtual servers. Virtual servers are formed by one or more large servers being subdivided into "logical" servers and the arrangement then being able to be operated as if it were a plurality of physical servers. The software used for this purpose is referred to, for example, as VmWare. In a corresponding exemplary embodiment of the invention, a large LINUX server drives a number of WINDOWS emulations via VmWare. The individual operating systems are monitored in exactly the same manner and, if appropriate, are moved to another virtual server, as would be the case with a plurality of physical servers. It is also possible to configure the system in such a manner that a computer is first of all restarted before the movement operation. This is of particular interest in virtual computers. If sufficient VmWare servers are not available, the "normal" movement path can be followed or a less important image is displaced and VmWare is started instead in order to be able to move the virtual servers which have failed or have been overloaded. List of reference symbols 1, 2, 3, 4 Computers 5 Monitoring computer 6 Lines 11 . . 19 Method steps OS1, 0S2, 0S3 Operating systems A1, A2, A3, A4 Applications D1, D2, D3, D4, D5 Services We Claim: 1. A method for operating a server farm having an arrangement of a plurality of computers (1,2,3,4) which are set up to execute a multiplicity of software units (A1..A4, D1...D5) and an additional monitoring computer (5) for monitoring the computers (1,2,3,4) and for intervening in the computers (1,2,3,4), and the status of the computers (1,2,3,4) being repeatedly monitored, characterized in that the software units (A1..A4, D1..D5) are assigned a weightage (L2K,SL) in accordance with their importance, and the following method steps are carried out in the event of a computer (1,2,3,4) failure: - information regarding which of the computers (1,2,3,4) have failed at the monitoring time and which software units (A1..A4, D1..D5) are affected thereby is collected, - a multiplicity of, i.e. at least two, possible transfer scenarios for transferring the software units (A1..A4, D1..D5) which are affected by the failure or failures to other computers (1,2,3,4) is determined, a transfer scenario respectively indicating which software units (A1..A4, D1..D5) are transferred to which computers (1,2,3,4), - the transfer scenarios are assessed using the weightage (L2K,SL) of the software units (A1..A4, D1..D5), - a transfer scenario is selected on the basis of the assessment, and - the selected transfer scenario is carried out. 2. The method as claimed in claim 1, wherein the software units (A1..A4, D1..D5) which are affected by the failure of a computer (1,2,3,4) are transferred to a reserve computer (4) with sufficient free capacity. 3. The method as claimed in claim 1, wherein the process of determining a transfer scenario comprises termination of the active software units (A1..A4, D1..D5) in order to create free capacity for transferring software units (A1..A4, D1..D5) from computers (1,2,3,4) which have failed. 4. The method as claimed in claim 3, wherein the weightage (L2K,SL) is effected using a plurality of attributes, a first attribute (license to kill) specifying a measure of the power to have other active software units (A1..A4, D1..D5) terminated in order to create free capacity and a second attribute (shutdown limit) specifying a measure of the resistance to the request for termination by other software agreements in order to be transferred from a computer (1,2,3,4) which has failed. 5. The method as claimed in one of claims 1 to 4, wherein the attributes of the software units (A1..A4, D1..D5) which are active in the computer arrangement and of the software units (A1..A4, D1..D5) which are affected by the failures are taken into account in order to determine a transfer scenario which is optimal overall. 6. The method as claimed in claim 5, wherein the possible transfer scenarios are first of all sorted in accordance with predetermined criteria before the optimal transfer scenario is determined from the possible transfer scenarios. 7. The method as claimed in claim 5 or 6, wherein a greedy algorithm is used to determine the optimal transfer scenario from the possible transfer scenarios. 8. The method as claimed in one of claims 1 to 7, wherein the software units (A1..A4, D1..D5) each comprise an operating system and applications. 9. An arrangement of a plurality of computers (1,2,3,4) for executing applications and a monitoring computer (5) for monitoring the computers (1,2,3,4), characterized in that the arrangement is set up to carry out the method as claimed in one of claims 1 to 8. ABSTRACT TITLE: A method for operating a server farm The invention related to a method for operating a server farm having an arrangement of a plurality of computers (1,2,3,4) which are set up to execute a multiplicity of software units (A1..A4, D1...D5) and an additional monitoring computer (5) for monitoring the computers (1,2,3,4) and for intervening in the computers (1,2,3,4), and the status of the computers (1,2,3,4) being repeatedly monitored. The software units (A1..A4, D1..D5) are assigned a weightage (L2K,SL) in accordance with their importance, and the following method steps are carried out in the event of a computer (1,2,3,4) failure: information regarding which of the computers (1,2,3,4) have failed at the monitoring time and which software units (A1..A4, D1..D5) are affected thereby is collected, a multiplicity of, i.e. at least two, possible transfer scenarios for transferring the software units (A1..A4, D1..D5) which are affected by the failure or failures to other computers (1,2,3,4) is determined, a transfer scenario respectively indicating which software units (A1..A4, D1..D5) are transferred to which computers (1,2,3,4), the transfer scenarios are assessed using the weightage (L2K,SL) of the software units (A1..A4, D1..D5), a transfer scenario is selected on the basis of the assessment, and the selected transfer scenario is carried out. {FIG. 4}

Documents:

01930-kolnp-2006 abstract.pdf

01930-kolnp-2006 claims.pdf

01930-kolnp-2006 correspondence others.pdf

01930-kolnp-2006 description (complete).pdf

01930-kolnp-2006 drawings.pdf

01930-kolnp-2006 form-1.pdf

01930-kolnp-2006 form-2.pdf

01930-kolnp-2006 form-3.pdf

01930-kolnp-2006 form-5.pdf

01930-kolnp-2006 international publication.pdf

01930-kolnp-2006 international search report.pdf

01930-kolnp-2006 pct form.pdf

01930-kolnp-2006 priority document.pdf

01930-kolnp-2006-correspondence others-1.1.pdf

01930-kolnp-2006-correspondence-1.2.pdf

01930-kolnp-2006-form-26.pdf

01930-kolnp-2006-international search authority report-1.1.pdf

01930-kolnp-2006-priority document-1.1.pdf

1930-KOLNP-2006-(29-05-2012)-CORRESPONDENCE.pdf

1930-KOLNP-2006-ABSTRACT.pdf

1930-KOLNP-2006-AMANDED CLAIMS.pdf

1930-KOLNP-2006-CORRESPONDENCE.pdf

1930-KOLNP-2006-DESCRIPTION (COMPLETE).pdf

1930-KOLNP-2006-DRAWINGS.pdf

1930-KOLNP-2006-EXAMINATION REPORT.pdf

1930-KOLNP-2006-FORM 1.pdf

1930-KOLNP-2006-FORM 18.pdf

1930-KOLNP-2006-FORM 2.pdf

1930-KOLNP-2006-FORM 26.pdf

1930-KOLNP-2006-FORM 3 1.1.pdf

1930-KOLNP-2006-FORM 3.pdf

1930-KOLNP-2006-FORM 5 1...pdf

1930-KOLNP-2006-FORM 5.pdf

1930-KOLNP-2006-GRANTED-ABSTRACT.pdf

1930-KOLNP-2006-GRANTED-CLAIMS.pdf

1930-KOLNP-2006-GRANTED-DESCRIPTION (COMPLETE).pdf

1930-KOLNP-2006-GRANTED-DRAWINGS.pdf

1930-KOLNP-2006-GRANTED-FORM 1.pdf

1930-KOLNP-2006-GRANTED-FORM 2.pdf

1930-KOLNP-2006-GRANTED-SPECIFICATION.pdf

1930-KOLNP-2006-OTHERS 1..pdf

1930-KOLNP-2006-OTHERS.pdf

1930-KOLNP-2006-PETITION UNDER RULE 137.pdf

1930-KOLNP-2006-REPLY TO EXAMINATION REPORT 1...pdf

1930-KOLNP-2006-REPLY TO EXAMINATION REPORT.pdf

abstract-01930-kolnp-2006.jpg