BACKGROUND OF THE INVENTION Colleges, universities and other institutions of higher education with top-ranked schools of study can generally only accept a limited number of students during each tri-mester, quarter, semester or other term, etc. Moreover, many potential students generally cannot attend on-campus classes traditionally offered due to conflicts with work hours and family commitments, etc. These factors have prevented a large number of students and potential students from obtaining high-quality educations from top-ranked schools in various fields of study, including computer science. It is becoming increasingly important, however, for such institutions to offer its students top-quality courses in all disciplines, including courses covering state-of-the-art technologies. However, it is also expensive to provide such courses. In particular, the expenses for course development and for qualified professors and instructors can be high. To alleviate some of these expenses, automated learning or tutorial systems have been developed.

With respect to computer science and by way of an example only, the heretofore existing system of higher education has not been pervasive enough to provide a quality, cost-efficient education to all potential students.-Some of the most effective software development concepts and techniques, such as used in object-oriented software development, have been slow to be fully utilized due to a significant overhead in providing courses covering the latest concepts in software development.

Accordingly, it would be advantageous to a system and method for providing and/or supporting courses to be offered at offering institutions that reduces the costs associated with course development and distribution.

SUMMARY OF THE INVENTION In general, the present invention preferably comprises systems and methods that enable an institution of higher learning, such as a college or university, to provide and/or support courses offered at other"offering"institutions. The institution making the courses available and providing support to the offering institutions which offer the same may or may not have been involved in the development or creation of such courses.

In a presently preferred embodiment, the teaching systems and methods of the present invention high-quality courses, including computer science courses, created by a developing institution or provided

by a supporting institution are delivered via a state-of-the art database system via the World Wide Web portion of the Internet to offering institutions supported by mentors assigned by the developing or supporting institutions. The courses made available through the teaching system of the present invention are designed to be highly focused, hands- on experiences. All course material is preferably delivered via the Web, with supplemental readings assigned from specified textbook (s) throughout the course. While all course material is delivered or assigned via the Web, the curriculum is not intended for use in a strictly distance- learning environment, where students progress through the material at their individual pace. Because of the rigorous nature of the courses, instructors, certified by the developing institution or supporting institution, who teach the courses at offering institutions are relied upon to a significant degree.

Preferably, the courses of the teaching system and method of the present invention are modeled on semester length college courses that most faculty at offering institutions and their students find to be highly focused and challenging. Preferably, students spend between 120-150 hours on each course, broken down as follows: 20 hours of lectures (maximum of 50% in-class time); 20 hours of instructor-led labs and hands on activities (minimum of 50% in-class time); 40-60 hours of hands on time with PC outside of lab or in lab; and 40-50 hours of self-study of course materials (on the Web, in textbooks or excerpts and class notes).

The courses offered via the teaching system of the present invention are designed to help orient students to problem solving. These courses require a different model of course delivery than traditional classes with similar subject matter. Student preparation before each class is a significant ingredient to the effectiveness of the courses and, therefore, students preferably are required to read all assigned Web and text content and take any quizzes via the system prior to each class session. All assessments or tests preferably are taken by students while logged onto the system of the present invention. Students must receive permissions to take all practical assessments (written tests or exercises).

The practical assessments preferably are proctored by an instructor.

Upon completion, a practical assessment must be uploaded to the developing or supporting institution's server to be submitted for grading.

Failure to perform the uploading process properly blocks a student's further access to the system. In addition, a student may not access any other part of a course while an exam is being taken.

In accordance with the present invention, the preferred use of face-to-face class time involves the instructors helping students to learn to perform the tasks of the course. To encourage students to complete all assignments prior to coming to class, quiz grades are preferably recorded before the assigned class and such quiz scores are used in the computation of final grades. Preferably, class time should not be spent by the instructor summarizing and spoon-feeding Web and textbook content to students. Instead, it is preferred that the instructor answer questions,

clarify concepts, give additional examples, coach students to complete exercises, direct students to find resources and answers to their own questions, and help the students master the assigned materials and tasks.

Preferably, no more than fifty percent of the duration of the class sessions is spent in traditional lecture, and a minimum of fifty percent of the duration of the class sessions is spent on providing students with hands- on activities and help with exercises, in view of the fact that students often need guidance to define problems and to develop strategies to effectively and thoroughly solve the course assessments.

The teaching systems and methods of the present invention preferably comprise the use of mentors employed by a developing institution or a supporting institution which provides a course or courses to be offered at the offering institutions. At least one mentor is assigned to a plurality of offering institutions to remotely monitor and enhance, preferably via the Internet, the performance of instructors and students at the respective offering institutions assigned to the mentor. The role of the course mentor is preferably one of support and guidance. The mentors are trained and have the experience to provide support and assistance to instructors throughout course preparation and delivery. The mentors assist in areas such as course content, technical support, course delivery and guidance, and various other types of assistance as instructors progress through a course. Course mentors will also alert instructors to errors in course content, and to portions of the course where students frequently encounter problems. Thus, the mentors of the developing or

supporting institution help the instructors at the offering institutions to master the material of the courses and to develop confidence in the use of the course software and the web site of the developing or supporting institution. Throughout the process of becoming an instructor certified by the developing or supporting institution and while teaching a course, the mentor guides and provides assistance to instructors, as is necessary, to ensure the successful delivery of the course.

Students and instructors preferably communicate with the course mentor via the Internet, but may also do so by telephone or facsimile if any system problems occur. The mentor normally provides notification as soon as the problem is resolved, or if there will be a delay in resolving the problem with the system.

Instructors and offering institutions also contact the mentor for any other support required or for questions that arise. In a preferred embodiment of the system, the instructors and offering institutions alert the mentors to problems or questions so that the developing or supporting institution is better able to resolve the same and to help other instructors teach the courses efficiently. Because of the relationships between the developing or supporting institution and the offering institutions, and because assistance is provided to instructors as they deliver the course materials, the developing or supporting institution can provide guidance and assistance to instructors for developing course schedules and syllabuses to make sure the course material is comfortably covered in the term allotted (e. g., a semester, quarter, etc.). Preferably, all quizzes,

exercises and assessments have explicit due dates so students don't fall behind schedule.

Preferably, the developing or supporting institution assists the offering institution with lesson preparation for the courses. In addition to providing support to instructors with course materials, the developing or supporting institution also preferably provides suggested classroom activities and additional hands-on practice exercises for students. All such assistance is preferably provided through the course software and through the mentors. Instructors will preferably reciprocate by providing the developing or supporting institution with any exercises or related labs that the instructors developed that might help students master concepts presented in a course.

When possible, the mentors preferably provide instructors with course outcomes for each course to allow instructors to determine what concepts to emphasize during class lecture time. In a preferred embodiment of the teaching system and method of the present invention, the course outcomes are reviewed prior to the start of each course, and periodically throughout the duration of the course. The course outcomes provide a comprehensive list of concepts and skills the students should have mastered upon course completion. The use of the outcomes helps instructors to focus lectures and to develop classroom activities to reinforce such concepts.

Because of instructors'relationships with students and the experiences in the classroom, all instructors are in a unique position to provide the developing or supporting institution with feedback regarding the course materials, including Web content, textbook materials, and support from the mentors. As such, the instructors preferably are encouraged to contact the developing or supporting institution mentor at anytime to discuss any concerns, suggestions or questions regarding any aspect of teaching the courses.

Preferably, the developing or supporting institution is committed to providing the highest quality courses possible. Faculty at offering institutions are significant components to the quality assurance process and all relevant feedback is used by the developing or supporting institution to revise and improve the courses. As the instructors progress through the courses, errors in the course materials, suggestions for different ways of presenting concepts and organizing the course contents, items to be included into or omitted from the course materials will likely be identified. All feedback regarding course revision and improvement is used by the developing or supporting institution, which preferably revises each course twice annually.

Other features and benefits of the present invention will become apparent from the detailed description with the accompanying figures contained hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an embodiment of a developing or supporting institution home page where students and instructors can log into the system of the present invention; FIG. 2 is an embodiment of a login screen for logging into the system of the present invention; FIG. 3 is an embodiment of a course introductory page for a course offered via the system of the present invention; FIG. 4 is an embodiment of a logout screen displayed upon successfully logging out of the system of the present invention; FIG. 5 is an embodiment of a help screen used by students and instructors for obtaining help in using the course software of the system of the present invention; FIG. 6 is an embodiment of a student record page generated by the system of the present invention; FIG. 7 is another embodiment of a student record screen generated by the system of the present invention; FIG. 8 is an embodiment of a course outline page showing a first level of detail of a course outline generated by the system of the present invention;

FIG. 9 is an embodiment of a course outline page showing a second level of detail of a course outline generated by the system of the present invention; FIG. 10 is an embodiment of a course outline page showing a third level of detail of a course outline generated by the system of the present invention; FIG. 11 is an embodiment of a"course activity blocked" message generated by the system of the present invention; FIG. 12 is an embodiment of an"access blocked"message generated by the system of the present invention; FIG. 13 is an embodiment of a main menu of the instructor staff interface of the system of the present invention; FIG. 14 is a block diagram illustrating the relationship between an instructor and the sections of a course taught by that instructor and the students in the sections taught by the instructor via the systems and methods of the present invention; FIG. 15 is an embodiment of a listing, generated by the system of the present invention, of all currently available courses for which an instructor has staff access; FIG. 16 is an embodiment of a histogram for a multiple choice quiz generated by the system of the present invention;

FIG. 17 is an embodiment of a message facility page generated by the system of the present invention; FIG. 18 is an embodiment of a student details page generated by the system of the present invention; FIG. 19 is an embodiment of a student message log page generated by the system of the present invention ; FIG. 20 is an embodiment of a student activity history page generated by the system of the present invention; FIG. 21 is an embodiment of a page generated by the system of the present invention indicating those students who have not done any work in a particular module of a course; FIG. 22 is block diagram illustrating the relationship between the main menu functions of the staff interface page, the information accessed through those functions and the various paths to the feedback page for a student containing feedback and grades therefor.

FIG. 23 is an embodiment of an all grades page generated by the system of the present invention; FIG. 24 is an embodiment of an ungraded assessments page generated by the system of the present invention; FIG. 25 is an embodiment of a student roster page generated by the system of the present invention;

FIG. 26 is an embodiment of a course module list page generated by the system of the present invention; FIG. 27 is an embodiment of a module details page generated by the system of the present invention; FIG. 28 is an embodiment of a task list page generated by the system of the present invention; FIG. 29 is another embodiment of a student record page generated by the system of the present invention; FIG. 30 is an embodiment of a grader downloader tool page generated by the system of the present invention; FIG. 31 is another embodiment of an ungraded assessments page generated by the system of the present invention ; FIG. 32 is an embodiment of a window displaying the contents of a grader folder during the process in which an instructor downloads assessments to be graded from the network server node to a client server node of the present invention; FIG. 33 is an embodiment of a window displaying an arrangement of folders within a"Grader"directory after an instructor has downloaded assessments to be graded from the network server node to a client server node of the present invention;

FIG. 34 is an embodiment of a window showing the directory hierarchy created inside a"Grader"directory after an instructor has downloaded assessments to be graded from the network server node to a client server node of the present invention; FIG. 35 is an embodiment of a feedback page generated by the system of the present invention; and FIG. 36 is an embodiment of an"edit feedback page" generated by the system of the present invention.

DETAILED DESCRIPTION OF THE INVENTION Preferred embodiments of the present invention are illustrated in the FIGURES, like numerals being used to refer to like and corresponding parts of the various drawings.

In a preferred embodiment of the present invention, all the courses, including course schedules, are delivered via the World Wide Web using the developing or supporting institution course software. By using this system, the developing or supporting institution is able to continuously provide the most up-to-date information to all students and instructors. Preferably, the system resides in a secure site that relies on the use of cookies for security. Any browser used to access the developing or supporting institution courses must be Java-enabled and be configured to accept cookies. In a preferred embodiment, two levels of access are available to instructors at offering institutions. The default user access is

"Student". The system allows each person to be a student in one and only one section of each course. Student access provides complete access to course materials and provides the same access as actual students have.

Instructors are also given"Staff Access", which provides the necessary interface to view student records, grade students, and locate tools and resources helpful to instructors. Instructors can have staff access in multiple sections of any specific course. Prior to teaching courses, instructors preferably work through the course materials as their students would. By working through the materials and experiencing the courses as a student, instructors are in a much better position to assist and to support students.

Because the developing or supporting institution courses are on a secure Web site, each user must log in every time he or she wishes to access a course. Preferably, each user's web browser is Java-enabled and comprises Internet Explorer 4.06 or a higher version of any of these browsers. Also, each user may only be logged in to one course at a time; logging into a new course causes the system to automatically discard any previous authentication. Each individual has only one account for all the developing or supporting institution courses, so the username and password is the same for all courses in which the user is enrolled; changing the password in one course changes it for the entire system, and, as a result, the password for all courses is changed. As shown in Figure 1, a user logs into the system by going to the developing or supporting institution's Home Page and using the drop down arrows at the bottom of the page, selects the

correct site from the Site drop down box 12, and the correct course from the Login drop down box 14 and clicks OK 16. The login screen 18 is then displayed as shown in Figure 2. The user then types his username and password in the appropriate boxes 20 and 22, respectively, which are preferably case sensitive. Afterward, by clicking the Login button 24, the system authenticates the username and password, retrieves the user's information, and displays the course introductory page 26 (Figure 3) is displayed.

Preferably, when finished using the developing or supporting institution's web site, the user always logs out to make sure that his/her authentication has been properly discarded so no one else can access that user's account information. To logout of the system, the user clicks the Logout hot button 28 in the upper right corner of the screen. A Logout Successful screen 30 (Figure 4) is then displayed.

The first time a user logs into a course, he/she is preferably presented with a multiple-choice pre-test specific to that course. This test is not intended to be used for grading or evaluation purposes, but is designed to help the developing or supporting institution continuously improve all courses. No other parts of the course may be accessed until the pre-test has been completed and submitted. Unlike other multiple-choice assessments, the developing or supporting institution web site does not provide feedback or a score for pre-tests.

The course introduction page 26, as shown in Figure 3, is the first page to display upon a successful login to a course. This page briefly describes the activities a student can perform with the developing or supporting institution course software, as well as provides a brief course description and textbook requirements for the course. The left-hand column of the course introduction page 26 provides a web-based outline 34 for the course, which can be used to access course materials.

As shown in Figure 5, the help page 36 of each course preferably contains general help topics 38 including downloading and using a correct web browser, navigating the courses, using the student record, using the course message facility, taking exams and quizzes and contacting the developing or supporting institution. Additionally, course- specific help may also be provided in some courses. The help page 36 is accessed by clicking on the"Help"hyperlink 35.

The Student Record page 40 (Figure 6) provides an overview of a student's progress through the course as well as access to course- specific information such as the Instructor's email 42 and section specific information. Access to the Student Record page 40 is obtained by clicking the Record hyperlink 39 in the upper right corner of the screen.

Email can be sent to the instructor of the section by clicking on the instructor's name 42. The Section hyperlink 44 provides access to any course-specific information, such as class day and time and instructor contact information. Other information, such as class notes, syllabus or

class schedule can be sent to the mentor to be posted for students to access. Instructors having their own web site can also maintain this information thereon by sending the course mentor the URL to be included in the section link 44 for students to access.

Preferably, various types of assessments or tests are used in connection with the methods and systems of the present invention including traditional limited response tests and quizzes, such as multiple choice tests, true or false tests, and fill in the blank tests, and open-ended response tests such as essay responses. Throughout this specification, assessments or tests that require open-ended responses are referred to as either practical assessments or constructed assessments. Constructed responses and/or practical responses are not limited to written text, but may include graphics, videotaped performances, audio responses, as well as other forms of responses.

As shown in Figure 6, the feedback of any assessment that has been taken can be viewed by clicking the number grade 46 in the Grade column 45. Any assessment that has not been taken can be taken by clicking the NT link 48 in the Grade column 45. The course outline 34 in the left-hand column 32 is preferably used to re-take any assessments.

Assessments that have already been submitted cannot be re-taken through the Student Record page 40.

The Message facility is preferably used by all instructors and students to discuss course issues. Only the instructor and the developing

or supporting institution staff, including the mentor have access to view communications from students to protect the privacy thereof. Students can only send messages to the instructor of their section. The message facility is accessed by clicking on the"Message"hyperlink 50 in the upper right corner of any course page to display a message log 100. If necessary, the scroll bars can be used to view messages lower on the screen. The message can then be typed in the Message text box 97. After the message is typed, clicking the Submit hyperlink 53 sends the message and the page refreshes with the new message displayed. The Message log is intended to facilitate communication between students and instructors, and designed to accept only text in either ASCII or HTML format. Preferably, it should not be used to transfer large amounts of data, such as exercise submissions.

Each course has an outline 34 which can be displayed in one of three levels of detail. Level 1 (Figure 8) displays only the course unit numbers 56 and titles 54. Level 2 (Figure 9) displays course unit numbers 52 and module numbers 56 and titles 54 as well as exams 58. Level 3 (Figure 10) displays every element of a course, including corresponding course Web pages 60, exercises 62, quizzes 64 and practice quizzes 66. outlines 34 can be used to navigate the course contents. The course outline is viewed by clicking on the Outline hyperlink 33 in the upper right corner of the screen. By default, the Outline link 33 at the top right of the screen displays the Level 1 outline. The Level 1 outline (Figure 8) displays only the Unit numbers 52 and titles of a course 54, with links to

each Unit introductory page 26. Access to a more detailed outline is obtained by clicking on the desired level of content hyperlink 31. The Level 2 outline (Figure 9) displays the Unit numbers 52, Module numbers 56 and titles of the course 54, with hyperlinks 51 that provide access to the corresponding web pages of the course. The Level 3 outline (Figure 10) displays all pages of the course, including section numbers 60 and page title 61. Access to any of the pages is obtained by clicking on the appropriate hyperlink.

Preferably, two types of assessments are used in the teaching system and methods of the present invention, multiple-choice assessments and practical assessments. Multiple choice assessments typically comprise ten randomly generated multiple-choice questions and include multiple choice quizzes and exams. Upon the submission of any multiple choice assessment, the course software automatically grades the assessment and provides the student with immediate feedback (see Figure 35) for each question, including the location of the correct answer in the assigned text or Web content. Multiple choice assessments must be submitted once started, or the course software will block the student from accessing any other part of the course. (See Figure 11).

Practical assessments require the demonstration of skills set forth in the related course material, such as writing a small program or creating a presentation, and include exercises, practical quizzes and practical exams. Practical assessments must be uploaded and submitted in order to be available to the instructors for grading. The uploading

process places a copy of the file (s) onto server of the developing or supporting institution, but does not allow the instructor to access them.

Submitting files puts the files in a location where instructors can access and grade a student's work contained therein. Failing to complete both steps for any practical quiz or exam results in the student being blocked from all other parts of the course until completing the upload and submit process. Practical assessments are graded by the instructor using the appropriate rubric.

Quizzes and exercises may be taken at any time by the student and do not need to be proctored. Quizzes and exercises can also be re-taken by students up to five times through the course outline 34.

Trying to re-take through the Record page 40 will display the feedback of the original submission and will not give students access to re-take the assessment. In a preferred embodiment of the present invention, the instructor is required to turn on system permissions in order for a student to access and take an exam which is proctored. If permission is not activated, an error message is displayed as shown in Figure 12. Exams may not be re-taken without the consent of the developing or supporting institution mentor.

All quizzes and exams are preferably timed assessments, and should be completed in the designated time in order for students to receive full credit. Exercises have no time restrictions for completion. Preferably, multiple choice assessments are allotted 30 minutes, and practical portions are generally allotted 45 minutes, although this may vary by

course or due to special circumstances. The course software does not provide any message regarding the elapsed time during the progress of an assessment until the assessment has been submitted.

Although spaces in filenames are permitted, the course software automatically truncates filenames with spaces by eliminating all characters to the left of the first space after the file extension. Because of this, students are instructed to name their files without any spaces in the name.

Once a timed assessment (any quiz or exam portion) has been started, no other part of the course may be accessed until the assessment is completed. Trying to access any part of a course after being blocked will result in an error message as shown in Figure 11. Once a student is blocked, the only way for a student to become unblocked is to complete the upload and submit process for the assessment. In the event the assessment was started accidentally, the student must submit the multiple choice file or minimum number of required files for the assessment. However, the student may elect to submit a blank file and use the message facility to communicate with the instructor so the grade for the assessment is not recorded and used for the course grade. To access the assessment that is causing the block, the student must go to the Record page 40 and click the"restart your work"hyperlink 41 to go to the assessment in progress.

The developing or supporting institution provides instructors with various tools to help them perform their teaching duties using the system of the present invention. For each section in which an instructor has Staff permissions, a list of functions is available to provide access to various activities needed to effectively teach and support students.

Additional tools may be provided for some courses by the course mentor.

As shown in Figure 13, the staff interface 70 is designed for the instructor to communicate with students and staff, grade and provide feedback for practical assessments and tasks, monitor student progress and manage students. Courses are broken down into components or units 52, modules 56, pages 61 and graded tasks 68. Tasks 68 are based on course material and include exercises, quizzes and exams. Practical assessments are graded by instructors, and randomly generated multiple- choice sections are graded by the system software.

All instructors have staff permissions in the section (s) they teach, and trainees are given access to the Staff Interface 70 after learning the Student Record page 40. The Staff Interface 70 of the teaching system provides access to student records, grades, student messages, and other teacher functions. Access to the Staff Interface 70 is determined when the user initially logs in to the system. For courses in which instructors have Staff level access, the necessary links and tools can be reached via the Record page 40 (see Figure 6) or the Staff link 69 in the upper right corner of the top frame.

Only instructors and mentors have access to staff level functions. Individuals without staff privileges do not have access to the Staff Interface 70 of the course. The Staff Interface documentation can be accessed on the Web or by clicking the INST section link 116 on the Student Record page as shown in Figure 29.

For individuals with Staff permissions, a Staff hyperlink 69 is displayed in the upper right corner of, for example, the course introductory page 26. For individuals having no Staff permissions, the Staff hyperlink 69 is not displayed.

The Staff Interface 70 may also be accessed from the course introductory screen 26 by clicking the Record hyperlink 39 in the upper right hand corner to access the student record page 40 upon which the"Go to Staff Interface"hyperlink 71 (Figure 6) is displayed for individuals who have Staff permissions in a specific course. Clicking on the hyperlink 71 brings up the Staff Interface 70.

As shown in Figure 13, the Main menu 72 of the Staff Interface 70 provides access to the various functions 73 and tools of course software specific to instructing a course. Clicking the"Ungraded Assessments"link 74 brings up a list of the tasks for all students yet to be graded for the current section. The"Messages"link 75, when clicked, allows for communication with a selected student or all students The "Student Roster"link 76, provides access to a list of all active students for that instructor. The"Course Module List"link 78 lists all modules in the

course. The"Task List"link 80 lists all tasks for all modules in the course. The"All Grades for Current Section"link 82 displays all grades for all tasks for all students in the section. The"All Grades for All My Sections"link 84 displays all grades for all tasks for all students in all sections for that instructor. The"Course Permissions by Student"link 86 turns on/off permission for students to take exams. The"Dropped Students"link 88 lists the students dropped from the current section. The "Staff Courses"link 90 lists the courses for which the user is an instructor.

The main menu 72 of the staff interface 70 can be accessed at any time by clicking on the"Main"link 79 at the top right of the screen.

The Staff Interface 70 may be exited by closing the browser window.

The links that are displayed on the Staff Interface page 70 provide access to information that is section-specific. Preferably, all active instructors will have access to three sections; an INSTRUCTOR section 43 for all instructors currently teaching a section of the course, a ZSOLUTIONS section 47 which contains all solutions and rubrics for the course ; and the course section they are currently teaching.

Students are grouped into sections within a course, with one instructor having primary responsibility for that section. Preferably, only one section can be worked with at a time. Sections are sorted alphabetically in the Section drop-down box 77, and upon entering the z Staff Interface 70, the section is set to the one closest to"A". Using the

section drop down box 77, an instructor can view the sections he/she is responsible for and change from section to section as required. The staff functions 73 on the Main page 72 of the Staff Interface 70 will apply to the section displayed in the"Current section is :" drop down box 77. Figure 14 is an illustration of the relationship between an instructor and the sections of a course for that instructor.

The Instructor section is the section for all instructors who are currently teaching a section of a course. This section provides student- level access to course materials. Preferably, only the developing or supporting institution employees have staff permissions in this section.

The message board in this section is preferably used by instructors to communicate with the course mentor regarding any course-related issues.

The ZSOL or SOLUTIONS section 47 provides solutions and rubrics for all instructor-graded assessments. Access to the solutions and rubrics is obtained by clicking the Ungraded Assessments link 74 from the main page 72 of the staff interface 70. Preferably, the instructors do not grade assessments in ZSOL section 47, since that would render any such assessments unavailable to other instructors.

Instructors have staff privileges for all courses they are currently teaching. Preferably, the sections are named according to the naming scheme of the offering institution in order to make it easier for instructors to keep track of students and sections.

Preferably, there are two types of sections used for training purposes. One section allows student-level only access, to provide an area for instructors working towards certification (from the developing or supporting institution) to work through the Web content and become familiar with the student interface course software. Only the developing or supporting institution employees have staff level access to this type of section, so instructor trainees and candidates can feel confident about submitting exercises and taking exams and quizzes without other instructors viewing their performance. Trainees and candidates also preferably use the message board to communicate with the mentor to assist them in mastering the course material.

The second training section provides staff level access to all instructor candidates, and provides instructor trainees the opportunity to experiment with the functionality of the Staff Interface 70. After instructors have had a reasonable amount of experience with the course software at the student level and have demonstrated proficiency in the course materials, staff level privileges are granted so that such instructor trainees can master the functionality of the Staff Interface 70.

Navigation among the various sections is accomplished by clicking on the Main link 79 at the top right of the screen. The Main page 72 of the Staff Interface 70 is displayed. Clicking on the Current section is: drop down box 77 in the upper left brings up a pop-up window (not shown) with a list of available sections. Clicking on the section desired in the pop-up window effects the change to that section whereupon the staff

functions apply to the newly selected section. An instructor can view a list of all currently available courses for which he/she has staff access (see Figure 15) by clicking the Staff Courses link 90 on the Main page 72 of the Staff Interface 70. Access to a specific course 91 is obtained by clicking the box in which that course is named whereupon all staff functions 73 apply to the selected course. As shown in Figure 16, histograms are a type of graph used to display the grade distribution for an entire section by task and are accessed through the Student Roster link 76, Task List link 80 or Course Module List link 78 of the Main page 72 of the Staff Interface 70. Access to the Details page 92 (Figure 18) for a specified student is obtained from the Main page 72 of the Staff Interface 70 by selecting the Student Roster 76 whereupon a list of all active students in the section is displayed (Figure 25). By clicking the desired student name, the Details page 92 for the specified student is displayed. Clicking the correct link in the Histogram cell 94 for the desired task displays a histogram for the selected task providing the instructor with an indication of how this student performed on the task compared with the other students of that section. Clicking on the"View Message Log"link 103 on Details page 92 brings up the log 100 (Figure 19) of all messages to and from the selected student, including messages sent to"All Student".

As shown in Figure 26, a list of all modules in the course is displayed by clicking the Course Module List link 78 on the Main page 72 of the Staff Interface 70. Clicking the View Students cell 96 for the

desired components 93 and modules 95 brings up a Module Details Screen 98 as shown in Figure 27.

As shown in Figure 28, the task list 99 displaying a list of all tasks 68 in the course is displayed by clicking the Task List link 80 on the Main page 72 of the Staff Interface 70. Clicking on the correct link in the Histogram cell 94 on either screen 98 or 99 brings up the desired histogram.

Messages can be sent via the message facility to an individual student, all students in a section, or all staff. Messages are kept in a log 100 which can be browsed. The Message Log 100 displays text as HTML and appearance of a message can be controlled using the HTML-tags in the Message Box 97. A message can be sent by clicking Messages link 75 from the Main menu 72 or by going to an individual student's record page 40 and clicking the Message hyperlink 50.

Messages can be sent to all students or staff of a section by using the Messages link 75 of the Main page 72 of the Staff Interface 70 page. An area 97 for sending messages and the log 100 of all messages sent for that section is displayed as shown in Figures 17 and 19.

A log 100 of all messages to All Students, All Staff, or to or from the instructor is displayed below the"Message :" box 97 on the Log page. Replies to any of the messages in the log can be sent by clicking the underlined name after"TO :" or"FROM :" in the log 100. That name will be displayed in the"To :" box as the recipient and after typing the message

in the"Message :" box 97 and clicking"Submit", the message will be sent to the selected recipient (s). The recipient can be changed to"All Staff', "Students in this section"or a specific student by clicking the"To :" box 102 and selecting the alternative desired and the message will be sent to the specified recipient (s).

As shown in Figures 18 and 20, the course software keeps a record of all student activity that can be viewed and monitored by all staff members with access to a section. The information displayed includes the dates and times the student logged in or out (Figure 20), and the specific components, modules and tasks accessed by the student during each 'session. This functionality allows instructors to see what pages or assessments a student has viewed, as well as a history of the student's log in activity. The History page 104 for a selected student can be displayed by clicking the View History link 89 from the Details page 92 of the selected student.

As shown in Figures 6,7 and 18, the Grade column 45 preferably contains one of the following values: Value Meaning NT The task has not been started NG The task has been completed by the student, but not graded a red number Failing grade a green number Passing grade ?partial

? inprogress The student has begun to upload and submit files for an assessment, but has not completed the task ? started The student has started an assessment, or at least looked at the page, but has not yet uploaded or submitted any files Instructors can view a list of students who have not performed any work for a particular module. Such a list can be accessed by clicking the Course Module List link 73 on the Main page 72 of the Staff Interface 70. By clicking the"Go"link 105 in the desired row of the Who Has Not Done Work column 85 on the Course Module List page 87 (Figure 26), a list of all students who have not done any work in the selected module is displayed.

Students fall into one of two categories: active or dropped.

Preferably, student status is determined by the offering institution's roster, not by actual student activity. An active student is a student who has access to the system and is enrolled in the course according to the information of the registrar of the offering institution. A dropped student is a student who has dropped or withdrawn from the section according to the offering institution's roster. A dropped student cannot access any section in the course from which he or she is dropped.

The Student Roster page 37, as shown in Figure 25, is accessed by selecting the Student Roster link 76 from the Main page 72 of the Staff Interface 70. Such action displays a list of all active students in the current section.

Clicking on the desired dropped student's name displays the Student Details page 92 for that student. Clicking the Dropped Students link 88 brings up a list of all dropped students. Clicking on the desired student's name displays the Details page 92 therefor. Although dropped students cannot access the section from which they are dropped, their student record remains intact and any assessments uploaded and submitted prior to them being dropped can be accessed through this function.

An instructor, without knowing the student's current password, may change the password of a student enrolled in a section in which the instructor has staff access. The instructor changes the student's password by clocking on the Change Password link 83 at the top right of the Details page 92 for the student. Such action brings up the Change Password screen from where the instructor can change the student's password by typing in the new password of at least six characters. The course software does not provide confirmation that the password has been changed when a Staff member changes the password for a student.

In a preferred embodiment of the present invention, the course software does not grant access to exams until a staff member explicitly turns on the required permissions. After the student uploads and submits the files, permissions are automatically turned off for the student. Exam permissions can be granted on a per student and per exam basis, or can be turned on and off for entire sections. If the correct

permissions have not been granted to a student, a warning message is displayed as shown in Figure 12.

An instructor or the mentor may change the state of exam permissions for an individual student by clicking the Course Permissions by Student link 86 on the Main page 72 of the Staff Interface 70. This action displays a list of students for the section listing permissions for each exam portion (multiple choice and practical) (see Figure 27). The permission state for a specific student for a single assessment can be changed by clicking the appropriate cell for the student and exam. The page refreshes, displaying the recent change in permissions. An instructor or the mentor can change the state of exam permissions for all students in a section by clicking the"View Student's Access Permissions" link (not shown). This action brings up a list of students for the section, listing permissions for each exam portion (multiple choice and practical).

Clicking the Turn On button in the appropriate column at the bottom of the page turns on the permission for all students in the section. The page refreshes, displaying the recent change in permissions. The permissions can be turned off for all students by clicking Turn Off button at the bottom of the page. Again, the page refreshes, displaying the recent change in permissions. A"Y"indicates permission granted, an"N"indicates permission revoked, and an"X"indicates the student has no more access to that exam portion.

As shown in Figure 18, graded tasks 68, or assessments, include exercises 62, quizzes 64 and exams 65. Multiple-choice quizzes

and exams are graded automatically by the course software, while exercises and practical quizzes and exams are graded by the instructor through the feedback page 49 using the appropriate rubric. Each graded task may have multiple instances, with a new instance being created when a student takes or retakes a quiz or exercise. Figure 22 shows the relationship between the Main Menu functions 73 and the feedback page 49, where written comments and a grade are entered for practical tasks.

Links 82 and 84 on the Main page 72 of the Staff Interface 70 are used to display the grades for: all students for all assessments for the current section by highest or most recent grade; or all students for all of the instructors sections by highest or most recent grade.

Figure 23 is an example of a portion of the All Grades page 108 that is displayed. Selecting the current section displays only one section. Selecting the"All my sections"option will display all other sections for the instructor that are available for the current course and semester. Clicking a specific student's name on the All Grades page 108 brings up the Details page for that student. Clicking a specific numeric grade 110 in the grade column displays the details for the assignment selected and allows the instructor to Edit Feedback. Clicking the"NG" link 112 in the grade column links to the Edit Feedback page 125 (see Figure 36) where the instructor can grade and give feedback for the assignment.

There are four views for grading from the Main page 72 of the Staff Interface 70. The Ungraded Assessments link 74 can be clicked to display the tasks not yet graded for the for all students in the current section. The Student Roster link 76 can be used to select a specific student for viewing or grading. The Course Module List link 78 can be used by staff to select the entire course module to view or grade. The Task List link 80 provides access for selecting a specific task to grade.

As shown in Figure 24, clicking the Ungraded Assessments link 74 displays all assessments that are waiting to be graded in a section, student name, module, task, instance and grade. The Grade is listed as" NG"link 112 since the task is not graded. Clicking NG 112 in the Grade column displays the page which shows the assessment and allows the instructor to grade through the Edit Feedback page 125. Clicking the student's name in the Name column displays the details page for the selected student. Right clicking the"Download XML"link 119 and selecting"Save target as..." (Internet Explorer) or"Save Link As..." (Netscape) saves the XML file to use with the Grader tool.

As shown in Figure 25, selecting the Student Roster link 75 displays a list of all active students in the current section. Clicking a specific student name displays all tasks and grades for that student on a Details page as shown in Figure 18. Clicking a numeric grade in the Grade column displays details for the assessment selected. Clicking NG in the Grade column displays an ungraded assessment for which the instructor can grade and give feedback. Clicking a link in the Module

column displays the students and assessments for that module. Clicking a link in the Histogram column 94 displays a histogram of class results for that task. The Redo function 107 in the Make Current column 106 will open another instance of an assessment and will display an"NT'in the grade column. Since there, preferably, are only a limited number of instances for each assessment, the mentor should be consulted prior to using the Redo function 107.

As shown in Figure 26, clicking on the Course Module List link 78 displays modules for components. Clicking the"Go"link 105 in the desired row of the"Who Has Not Done Work"column 85 displays a list of students who have done no work for the module, as shown in Figure 21.

Clicking the"Go"link 105 in the desired row of the View Students column 96 displays a list of students and assignments for that module.

Several options are available from the Module Details page 98. Clicking the appropriate link in the Name column 109 displays assessments and grades for a specific student not limited to the current module. Clicking a specific grade 110 in the Grade column 45 displays the results of a graded assessment. If the corresponding assessment is automatically graded, the feedback provided by course software is displayed. If the assessment is manually graded, the Edit Feedback page 125 is displayed and the instructor can use this page to enter or change feedback and the grade. Clicking the NG link 112 in the appropriate cell of the Grade column 45 provides access to the Edit Feedback page 125 to provide initial feedback and a grade for an ungraded assessment.

Clicking the"Go"link 105 in the Histogram column displays a histogram of the scores for the corresponding task.

Clicking the Task List link 80 on the Main page 72 of the Staff Interface 70 displays a list of all tasks within module and component as shown in Figure 28. Several options are available from the Task List page 99. The Grader Tool can be downloaded to each instructor's computer by clicking the INST link 116 from the instructor's Student Record page (see Figure 29). The"Grader"Downloading tool page 118 (Figure 30) is displayed and clicking the HERE! hyperlink 120 brings up the File Download dialog box. Selecting the"Save this program to disk" option and clicking OK displays a dialog box with a prompt for the location in which to save the file. Preferably, a Grader folder is selected as the folder in which to save the Grader tool. Clicking save displays a message that the file is being saved. When finished, a Download complete message is displayed.

To help instructors download all assessments waiting to be graded so they can be graded offline, the system of the present invention preferably includes a Grader Tool. The Grader Tool and the related files are compressed into a self-extracting file using WinZip. A link to download these files can be found by clicking the INST link 116 in the Student Record page 40 (see Figure 29). After the grader tool has been downloaded and extracted, the instructions for installing and using the tool can be found in the README. HTML file. A standard installation of the Java Development Kit (JDK) 1.2.1 or higher is required to use the grader tool.

The JDK can be downloaded from Sun Microsystem's Web site at http ://java. sun. com/products/jdk/1.2/download-windows. html, following the instructions therein.

After downloading the files and reviewing the student submissions, the grade and feedback can be entered into each student's Edit Feedback page 125 for the assessment, or the Automated Rubrics may be used to provide the feedback.

A preferred embodiment of the present invention includes a software package made available to each instructor for use in downloading sets of tasks or assessments that have been submitted for grading. The software package includes: extract. bat-The batch executable creates the directory hierarchy and extracts the submitted tasks into this hierarchy. extract (. pif)-PIF file for extract. bat information (needed for W9518).

* xml4jll14. jar-Java Archive to extract xml files.

XMLTesterl_2_4. jar-Java Archive for the &num extracting executable. readme. txt-This readme file in text format. readme. html-This readme file in html format.

images-directory containing images for readme. html.

The software Package is preferably unzipped to the C: \Grader\ directory (the default of the self-extractor) because of set paths in the extract. bat file. Each instructor can edit the extract. bat file and change the variables as necessary or convenient.

The Ungraded Assessments link 74 on the Main page 72 of the Staff Interface 70 displays a list of student submissions in a specified section that need to be graded. The Download XML link 122 on the Ungraded Assessments Page 124 (Figure 24) provides a way to download assessments to the C: \Grader\ directory for grading. The Java Software Development Kit fdkl. l or higher) must be installed on the computer, and the bin directory must be in the path statement of the computer. This can be checked by opening a command prompt, typing"java- ?" and pressing"Enter". If a usage list is obtained, then the files can be downloaded. If an error message is obtained, the jdk must first be installed from http ://www. javasoft. com, or the path must be set to include the jdk\bin directory.

Since there are two Extract files in the Grader folder, care must be taken to ensure that the showgradexml. html file is copied into the extract. bat file, and not the extract shortcut. Dropping onto the shortcut will not cause any harm, but no files will be downloaded. The grader (downloader) tool can be used by clicking on the Ungraded Assessments link 74 to display the Ungraded Assessments page 124

(Figure 31). Right clicking on the Download XML (save this link to a file) link displays Pop-Up Window 123. After clicking"Save Target As..."or "Save Link As..." (depending on the web browser used), My Computer or Explorer is opened to view the contents of the Grader folder whereupon the showgradexml. html file can be dragged and dropped onto the extract. bat file (see Figure 32). The instructor must be logged into the developing or supporting institution web site in order for the files to download the assessments. A warning message may be displayed, along with a message to ignore the warning message. As long as files are being downloaded, any such warnings should be disregarded. A Command prompt (MS-DOS) screen is displayed as the batch file downloads the necessary files from the developing or supporting institution server. This activity may take awhile to complete. When finished, a message displays that the download is complete. MS-DOS is then exited by typing"EXIT' and pressing"ENTER." Preferably, a folder structure similar to the one shown in Figure 33 will be been created in the Explorer view. As shown in Figure 34, the directory hierarchy will be created inside the C: \Grader\ directory. The directory structure starts with the course version, and then has subdirectories of each section. Inside the section directories, there are located unit directories, with the unit number and unit name as the name of the directory. From this point, the hierarchy matches that of the course: there are directories for each assessment in the appropriate place as it is in the course listed as subdirectories of the appropriate components,

modules, or pages in which they exist. Inside of the assessment will be directories for each student and instance of that assessment that they took. The name of the folders are based on the student's user identification and the instance of the assessment. For example, the first time a student with user identification"x"takes a particular exercise, in the appropriate exercise directory, there will be a directory called"x_100" with all of the files that student"x"submitted. If the student retakes the exercise, then there will be a directory called"x_200", and all of the files submitted the second time would be in this file.

The Grader (downloader) tool made available to instructors via the system of the present invention should work, in most applications, to allow instructors to download student files to a C: \Grader\ directory.

The Grader (downloader) tool may, however, be customized. The Grader (downloader) tool program first uses the first java. exe it finds in the path.

If the program does not work, or java. exe is not in the path, and it has been installed, then java. exe must be set manually to be in the path or one of four variables in the extract. bat file must be set manually as follows: TARGETPATH=C : \Grader\-sets the location where the student file submissions will be automatically downloaded; any path may be set, but this one works well.

XMLJARPATH=C : \Grader\-the location where the extraction program lives.

SETJDK=0-Instructs the program to use the first java. exe found in the path (0) or to use the path set in the JDKPATH variable in the extract. bat file (1).

JDKPATH=C : \jdkl. 2.2\-This is an example illustrating that this variable should be set to the path to the particular jdk on the instructor's computer.

The Edit Feedback function of the course software provides instructors access to students'submitted assessments, so that instructors can provide appropriate comments and grades to student work. Practical assessments (practical quizzes and practical exams) require instructors to grade them and provide feedback to students about the grade provided.

Multiple choice assessments are graded automatically by the course software, and instructors do not need to grade them or provide feedback.

The Edit Feedback functions can be accessed from several functions found on the Main page 72 of the Staff Interface 70. The feedback provided will display how many questions were on the assessment, how many were answered correctly, the total score, each question asked with the student's answer and whether the answer was correct, and where in the Web content or course textbook the answer can be found. Occasionally, there may be a need for an instructor to change the grade given to the student by the system. In such an isntance, the instructor preferably contacts the Mentor for the course to have the change made.

Exercises and practical sections of quizzes and exams are graded manually through the Edit Feedback function. The Edit Feedback page 125 provides access to enter comments as text feedback for assessments, assign a percent grade, set the assessments to pass or fail, and view the results of a student's quiz or exam. As shown in Figure 36, any comments about an assessment are preferably typed between the <pre> and </pre> tags in the feedback text box 126. All feedback is preferably typed as HTML using appropriate tags.

In a preferred embodiment of the present invention, adequate feedback to an assessment comprises an itemization of point deductions specifying for each point deducted the reasons therefor, an itemization of point additions (e. g. extra credit) and the reasons for such point additions, a percent grade and a pass/fail indication. After all applicable feedback has been set forth, the instructor submits the feedback to the system by clicking the submit button 128 whereupon the student can access it. Preferably, after the assessment has been graded, comments should exist in place of the"No Feedback"entry shown in Figure 36. The grade box 130 will contain a value between zero and 100%, and the radio buttons 132 may be set to pass or fail or Redo.

In a preferred embodiment of the system of the present invention, access to the Edit Feedback page 125 may be obtained from various other locations, including the Main page 72 of the Staff Interface 70, the Course Module List page 87, the Details page 92 for a student, the Task List page 99, the View All Grades page 108, and the Ungraded

Assessments page 111. As discussed above, all of these latter pages can be accessed from the Main page 72 of the Staff Interface 70.

From the Ungraded Assessments page 111 (Figure 24), access to the Edit Feedback page 125 is obtained by clicking the"NG"112 in the Grade column of the student whose assessment is to be graded.

From the Task List page 99 (Figure 28), the Edit Feedback page 125 is accessed by first clicking the Students link 113 in the View Students column to access the Module details screen 98 for the desired module.

Clicking the"NG"link 112 or the Percent Grade link 110 on the Module Details screen 98 brings up the Edit Feedback page 125 for the selected student. From the Course Module List page 87 (Figure 26), the Edit Feedback page 125 can be accessed by first clicking the"Go"link 105 in the desired cell of the View Students column 96 which brings up the Module Details screen 98 for the desired module. Here again, clicking the NG link 112 or the Percent Grade link 110 displays the Edit Feedback screen for the desired student. From one of the View All Grades pages (e. g. page 108 in Figure 23) available from the Main page 72, the Edit Feedback page 125 for a particular assessment for a particular student may be accessed by clicking the NG link 112 or the Percent Grade link 110 in the appropriate cell. The Edit Feedback page 125 can also be accessed by clicking on the desired student's name on the Student Roster page 37 (Figure 25) to display the Details page 92 (Figure 18) for that student and then by clicking the NG link 112 of the Percent Grade link 110 in the Grade column.

To assist instructors with grading assessments and to promote consistent grading across sections and offering institutions, the system preferably includes automated rubrics for each course. These rubrics allow instructors to grade their students using checkboxes, and also provide for standard feedback to students. After checking all appropriate checkboxes for a student, the grade is automatically calculated and the feedback for each individual item is automatically entered into a textbox for instructor review. The instructor can verify the information, and then submit the grade to the system, where it is recorded for the student to access via the system. The Automated Rubrics may be used in place of the Edit Feedback page 125 described above.

To use the Automated Rubrics of the system of the present invention, the Java Development Kit (JDK) 1. 1 or higher must be installed on the PC, and the jdk\bin directory must be in the PATH statement of the PC in order to use the Grader tool or the Automated Rubrics.

Access to the automated Rubrics is obtained via the system as follows: First, from the Main page 72 of the Staff Interface 70, the desired section is selected with the Section: drop down arrow : The page updates, and the links displayed provide access to the selected section.

Clicking the Ungraded Assessments link 74 displays an Ungraded

Assessments page 111. After right clicking the Download XML link 119 and saving it to the Grader folder on the c: \ drive of the client PC.

The Grader folder can be opened using My Computer or Windows Explorer. Upon dragging the showgradexml. html file onto extract. bat file in the Grader folder, an MS-DOS (Command Prompt) is displayed. A series of commands are displayed as the extract. bat file downloads assessments from the developing or supporting institution Web server to the local hard drive. When the extract. bat file has finished executing, MS-DOS is exited by typing exit at the command prompt and pressing"Enter".

Upon displaying the contents of the Grader folder, the assessments waiting to be graded are displayed in a directory structure that uses the following conventions: course number \ section \ module \ assessment \ studentsubmittal number \ feedback. Generally, Microsoft Explorer is more convenient to use for grading multiple students than"My Computer". After opening the file"give_feedback. htmr'located in the Feedback Folder by double clicking it, an HTML page is displayed in a Web browser with the exercise or practical assessment followed by a table of checkboxes. The instructor must be logged into the correct course in order for the file to open properly. After opening the assessment of the student desired to be graded, the desktop is arranged to display the give-feedback. html file and the student's assessment next to one another.

The assessment is then graded by clicking the appropriate checkboxes for the assessment. If necessary, changes to the point value or to the feedback

text may be made. Also, the blank areas of the instructor feedback rows may be filled in to add or subtract additional points for the assessment. A minus (-) sign must be included in the point column to subtract points and a plus (+) sign may be added to add points, or the number of points may simply be typed in. Upon clicking the Calculate Grade button at the bottom of the feedback table, the feedback and grade for the assessment are displayed in the feedback box. The instructor can then verify that the feedback is accurate and may also add or change text as appropriate.

Upon clicking the Submit Grade button, the grade and feedback are submitted to the developing or supporting institution web site, and the student can then access the grade via the system. The above-described grading process using the automated rubrics can be used for each assessment to be graded. This process can also be used to change the grade of an assessment that has already been graded. The revised feedback and grade will replace the existing information on the developing or supporting institution web server.

In a preferred embodiment of the present, it is also possible to download assessments that have already been graded. On the task list page 99 there is a"Download Graded"column that contains Get XML links (Figure 28). These links access a file named showgradedxml (as opposed to showgrade_xml in Ungraded Assessments). This allows for the downloading of an XML file that is used exactly as the XML in the"Show me what I have to grade"section. The difference is that the showgraded XML file contains a list of files for that particular section,

and the particular assessment that was clicked. Preferably, there is an option to choose the files to be downloaded for the particular assessment.

Because the name of the procedure is different, the default filename is different. Instead of showgrade-xml. html, it is called showgradedxml. html. This is dropped in the same extract. bat as before in the grader-downloader program. Then all submits for all graded assessments in that task are downloaded and put into the same file hierarchy as described above. The Staff Interface may be exited by closing the Web browser window displaying the staff interface. This action does not log the instructor out of the course. To log out of the system, the Logout link in the upper right corner of the course pages must be clicked, or the Web browser window must be closed to log out of the course and destroy-the authentication.

In a preferred embodiment of a method for offering a course curriculum by an offering institution, the requirements for student certification by the developing or supporting institution are a minimum score of 80 or better on all exam portions in a course. The developing or supporting institution does not consider quiz or exercise scores for certification purposes. Any student who is working towards certification and scores below an 80% on an exam portion must consult with to their instructor about re-taking the exam. A student who scores below an 80% on a single exam in a prerequisite course but earns certification in higher level course will be granted certification for the prerequisite course.

For maximum student success, it is preferred that all instructors who teach the developing or supporting institution courses possess proficiency in both course subject matter and use of the developing or supporting institution Web Interface. Having demonstrated proficiency in these areas, instructor candidates will be awarded the developing or supporting institution Certification, which is granted on a course-by- course basis. After instructors have been certified for a course, they may teach that course at any offering institution. Only students taking a class from a instructor certified by the developing or supporting institution will be granted access to the developing or supporting institution course material. Preferably, there are currently two levels of instructor certification, provisional certification and instructor certification (full certification). Instructor candidates demonstrate their readiness to teach the developing or supporting institution classes by providing evidence of relevant prior experience and participating in the Instructor Orientation Program offered by the developing or supporting institution.

To be granted provisional certification by the developing or supporting institution, instructor candidates must satisfy the following specific requirements. Instructor candidates affiliated with a particular offering institution must submit a statement of qualifications detailing any relevant prior experience to the offering institution to be forwarded to the developing or supporting institution; freelance instructors must submit a statement of qualifications to the developing or supporting institution. This statement preferably consists primarily of a brief

description of the candidate's qualifications, including: (i) relevant educational experience; (ii) computing experience, including programming languages, technical work experience, Internet experience, and any other relevant skills; and (iii) teaching experience, including a syllabus, course location and the capacity in which the course was taught (instructor, teaching assistant, lab assistant, etc.) for each relevant course.

The instructor orientation program of the present invention preferably includes orientation sessions and exercises concerning the subject matter of the courses in the curriculum and or using the developing or supporting institution's Web Interface. Upon the developing or supporting institution approval, instructor candidates are given access to the appropriate courses on the developing or supporting institution Web site and then begin working through the course materials. Instructor candidates initially enrolled as students in the courses to provide them with the opportunity to work through the course materials and to experience the developing or supporting institution Web site from a student's perspective. As instructor candidates work through this process, they preferably use the developing or supporting institution Web Interface to submit all course assessments and to communicate frequently with the developing or supporting institution. Instructor candidates are responsible for obtaining copies of any textbooks required for a course.

Instructors who have prior experience in teaching the subject matter of a developing or supporting institution course preferably must complete and submit a representative set of course assignments, to be

approved by the developing or supporting institution. Instructors preparing in subject areas new to them, work through all course assignments with the assistance of the developing or supporting institution, if necessary.

All instructor candidates preferably are required to participate in one full teacher orientation session prior to teaching a the developing or supporting institution course. A full session preferably entails a minimum of two half-day segments (each 4-5 hours) with approximately 3-to-4 full-days worth of course-specific assignments to be completed between the two days. Course specific assignments can be completed by the instructor on the developing or supporting institution premises or at remote locations. A full session provides candidates and instructors with the opportunity for : hands-on training in the developing or supporting institution course software and Web Interface; finding and using a variety of tools that support instructional duties; and personal and informal interaction with the developing or supporting institution course mentors and staff After instructor candidates have become familiar with the student component of the Web Interface, they will be enrolled as staff members in the course and will be given opportunity to practice grading assessments using the developing or supporting institution tools and guidelines.

Because the course software comprises a significant aspect in the success of students, instructors, and the developing or supporting institution, it is preferred for all instructors to gain proficiency in using the Web Interface for all aspects of a course, including submitting and uploading files, communicating via the message board, and navigating the course. Initial training is provided in the teacher orientation sessions, and additional assistance is preferably provided by the developing or supporting institution as necessary to help candidates become proficient in using the system.

To become a developing or supporting institution Certified Instructor for any course, all instructor candidates must: meet all requirements for the developing or supporting institution provisional certification for the specified course; and successfully instruct one complete offering of the specified course. successful course delivery is determined solely by the developing or supporting institution, on the basis of student success and overall satisfaction with instructor performance; and satisfactory working relationship with the course mentor and other developing or supporting institution staff Instructors who but lack the appropriate background for teaching a course can still obtain developing or supporting institution provisional certification by completing all assignments of the course.

During the process of completing the assignments and mastering the material of the course, instructors preferably should contact the developing or supporting institution's staff for assistance and support as

necessary. Instructors may also attend classes taught by colleagues at offering institutions.

Students may be added or dropped from a section by sending the required info to the developing or supporting institution course mentor by email. The information required to add students is the course name, Software Systems Development 1 (e. g., SSD1), section number (e. g.

AB123), student first name, last name, social security number and email address (if available). The minimum information required to drop students is first and last name, or social security number; however, more information, if available, is preferred. Upon receipt of the message, the accounts will be created in or removed from the appropriate section, and the developing or supporting institution course mentor will reply with a confirmation email that includes the new student's logon information.

The offering institution should preferably provide the developing or supporting institution and course mentor with a class meeting schedule and to ensure that email is quickly read and responded to, especially during the first week of classes.

Dropped students cannot access any section in the course software from which they are dropped. For this reason, instructors preferably are to use extreme care and check with the registrar of the offering institution prior to requesting that a student be dropped.

In a preferred method of the present invention, all student submissions are graded with sufficient feedback on a weekly basis for

courses that are fifteen weeks or longer, every three days for courses ten-fifteen weeks in duration, and every two days for courses of a shorter duration. Also, all practical exercises and practical quizzes preferably are graded with thorough feedback provided to the students at least 3 days prior to the scheduled exam for the unit (s) covered by such exercises and quizzes. This preferred guideline helps to ensure that both the instructors and the students don't fall behind as the course progresses, and that any mistakes or misunderstandings the students are having are quickly rectified.

At offering institutions, the developing or supporting institution courses preferably serve two purposes. One purpose is for the developing or supporting institution certification or for credit and a grade at the offering institution. The requirements for student certification by the offering institution preferably are a minimum of 80% on each exam portion (multiple choice and practical) in a proctored setting. The exams must be given exactly as they are stated in the course materials and graded according to the rubrics. The scale used to provide students with a grade at any offering institution is at the discretion of the instructor and/ or the offering institution. The developing or supporting institution can provide guidelines for a grading scale, if necessary.

Feedback for grading should preferably indicate specifically which item (s) were used for reducing or increasing a student's score with the number of points for each item. This information is provided in the rubric maintained on the system for each assessment.

As with grading, the particular weight assigned to each assessment is at the discretion of the instructor and/or offering institution. As a guideline, the various assessments preferably should be weighted so the course grade reflects student effort and understanding of the material.

In order to promote consistent grading at all offering institutions and with all instructors, the developing or supporting institution preferably should develop and provide rubrics for all exercises, practical quizzes and practical exams. Rubrics are tools which are designed to be guidelines for how many points to take off in the case of incorrect or incomplete responses and where to give credit for the specific details of each assessment.

The rubrics for each exercise and assessment can be accessed from the ZSOL section of the Main Staff Interface page and clicking the ungraded assessments link 74. Then from the Display page, clicking both the hyperlink to the assessment to be graded and the filename.

For most instructors, the use of a rubric is a new skill.

Because of that, the developing or supporting institution preferably provides as much assistance and training as necessary to make instructors feel comfortable using rubrics. The developing or supporting institution also preferably checks a random sampling of all grading against the appropriate rubric to ensure consistency across all the sections of a course.

Because exams are the sole criteria for determining certification by the developing or supporting institution, policies are preferably instituted to ensure that the same standards and criteria are used by all instructors at all offering institutions. Preferably, such policies are strictly followed by all instructors.

At least one week prior to an exam, instructors preferably should review the practical portion of the exam to identify any dependencies on previous assessments. If applicable, students should to be instructed (verbally in class and on the message facility) to bring any necessary files to the exam. Typically, the required files will be the student solution to a set of exercises or quiz practicals.

At least one copy of the multiple-choice exam and of the practical exam are preferably printed out prior to the scheduled exam day so the exam can still be offered as scheduled in the event of a network or other system failure.

In a preferred embodiment of the present invention, the developing or supporting institution requires all exam portions (multiple choice and practical) be proctored by the instructor or suitable substitute.

As a result, no student who is absent from class on a scheduled exam day should have exam permissions turned on. Also, all student submissions should be verified prior to any student's leaving the classroom upon completion of the exam. Verification is accomplished by accessing the Student Record page 42 for the individual student and verifying that

there is a grade for the multiple choice section of the exam, and a NG (for Not Graded) for the practical portion thereof. The submissions for several students can be checked by clicking the View All Grades link After a positive verification the exams have been properly submitted, all permissions for the exams just taken should be turned off and student and the student may leave. If the Practical Exam grade of the student's record shows a"? partial","? started", or"? inprogress", the student must log back on to the course and finish uploading and submitting the required files containing the student's exam submission.

Occasionally, students will have a legitimate excuse to miss class on a scheduled exam day. If the instructor and offering institution allow for make-up exams, the developing or supporting institution preferably will permit a make-up only if the exam is in a proctored setting and all other exam policies are followed.

Occasionally, students will perform poorly on an exam. In this event, and if the instructor and offering institution allow exams to be retaken, the instructor preferably must then contact the designated mentor for the specified course and discuss the matter. The developing or supporting institution mentor will then review the matter and provide a final decision on whether or not to allow an exam re-take. Under such circumstances, the developing or supporting institution preferably has the final authority on whether or not to permit a student or students to re- take an exam.

The developing or supporting institution may also preferably institute a policy that will allow students to re-take any exam for a specified fee.

All students and instructors will preferably use the Message facility functions of the course software all course-related correspondence.

The system of the present invention allows the developing or supporting institution to monitor the messages sent via the system and can thereby detect problems and concerns that arise during the course. In this manner, the courses and existing Help information can be continuously updated and improved. Thus, the usage of the course messaging facility allows the developing or supporting institution to support instructors at offering institutions as they support students. The developing or supporting institution preferably does not have access to personal voice mail and email in the event that an instructor has an emergency and cannot respond to student questions, but the developing or supporting institution can access the Message facility to help instructors support their students. The Messages facility is preferably checked by the developing or supporting institution every two days for ten-to-fifteen week courses, daily for courses of a shorter duration.

Any student requiring special requirements or accommodations to complete a developing or supporting institution course is preferably treated according to the offering institution's policies and guidelines. A mentor from the developing or supporting institution

preferably has experience and information concerning the accommodation of special needs students.

The mentors are preferably the primary contact and resource for the developing or'supporting institution courses and preferably provide guidance and support to instructors as they teach the developing or supporting institution courses. The specific help instructors can expect from the mentors preferably includes: guidance on developing a course schedule, syllabus and any class policies; answering content questions about the course material; holding regular conference calls or meetings with an open agenda for instructors teaching the same class; looking into any unaddressed issues; suggesting revisions to policies or procedures of the offering institution in offering a course or curriculum; regularly reviewing and monitoring the Message Facility of the system to pinpoint any problems with a course or it's content; randomly checking grading for consistency with the rubrics; notifying the instructors of course content that is particularly difficult for student; uploading course-specific information into an appropriate section of the developing or supporting institution Web Site; scheduling; counseling with respect to the developing or supporting institution's course policies; providing notification of related web pages and links to relevant resources; and other support to any issues that will help instructor master the course material, use the course software and support students.

Preferably, each mentor will have a thorough understanding of his/her role in helping the offering institution to offer the courses

developed by a developing institution or provided by a supporting institution. Each mentor will also have a thorough understanding and be well versed in the use of administrative section of the Staff Interface 70 of the system of the present invention. Each mentor will also have an intimate knowledge of and vast experience with the course materials.

Each mentor will also know how to use all the tools made available to the offering institutions in conjunction with the system of the present invention and provide support, including troubleshooting, in using the course software.

In a preferred embodiment of the present invention, the mentors train and conduct the certification of the instructors. The mentors follow, preferably over the Internet, the instructors'progress through each course that the instructors are training to teach. The mentors review the prospective instructors'submissions and provide feedback as appropriate. Also, the mentors schedule time to assist each trainee with the course materials. The mentors understand the purpose of the different sections that can be accessed through the course software and make sure that the appropriate level of access is provided for each instructor or instructor trainee. Prior to the beginning of a course term, the mentors will provide each instructor with the specific task or tasks each instructor must complete before beginning to teach. The mentors also preferably conduct the instructor orientations and provide each instructor trainee with sample assessments to grade with and without the use of the automated rubrics of the present invention The mentors also

ensure that the instructors have all of the latest tools, including the grader tool (requiring proper JDK\JRE installations), and that such tools are installed correctly. In addition to making sure that the instructors know how to use the message facility of the system of the present invention, the mentors provide secondary contact information so that the instructors can reach the mentors in the case of problems with the system.

Mentors are also preferably responsible for providing the instructors with suggested course schedules, course syllabi, course outcomes, guidelines for lectures, lab session plans, self-study session plans and other recommendations for students. Mentors also preferably collect contact information so that the instructors may be contacted either via the system or otherwise in case of system problems.

Preferably, the mentors will provide the instructors with support and guidance throughout the delivery period of each course and schedule regular conferences with instructors, preferably by telephone or videoconferencing. Each mentor also preferably checks regularly on grading practices to ensure that each instructor's grading is done in a timely fashion and is consistent across all sections of a course the instructor is teaching. The mentors also review the message boards of the system of the present invention to spot student issues and concerns, particularly common issues and concerns across different sections of a given course. The mentors also preferably encourage the students and instructors to use the message boards to facilitate the mentor's review thereof. The mentors also monitor the progress of the instructors and

students in a given course and whether the students and instructors are on schedule to complete the course in a given period of time. The mentors also preferably file regular reports with the developing or supporting institution and/or with other mentors to provide an update on the progress being made in each course and on any technical issues which need to be resolved within the system.

The developing or supporting institution will preferably prescribe the minimum course assessments to be completed by instructors having relevant experience prior to beginning to teach any of the developing or supporting institution courses. Instructors who are new to the subject matter preferably must complete all assessments in a course prior to beginning to teach that course..

Successful completion is preferably defined as achieving a minimum grade of 80% on each assessment. Multiple attempts at taking assessments are preferably permitted, and the developing or supporting institution preferably should provide assistance and support to instructors working to earn instructor certification.

The foregoing description of the invention has been presented for purposes of illustration and description. Further, the description is not intended to limit the invention to the form disclosed herein.

Consequently, variations and modifications commensurate with the above teachings, and the skill or knowledge in the relevant art are within the scope of the present invention. The preferred embodiment described

herein above is further intended to explain the best mode known of practicing the invention and to enable others skilled in the art to utilize the invention in various embodiments and with various modifications required by their particular applications or uses of the invention. It is intended that the appended claims be construed to include alternate embodiments to the extent permitted by the prior art.