e-Learning Case Studies

• Name and e-mail: Kevin Renfrew (K.C.J.Renfrew@bath.ac.uk), Roger Gardner (r.gardner@bath.ac.uk)
• Keywords (pedagogical): e-assessment, diagnostic, language testing
• Keywords (technical):
• Discipline: n/a
• Year / ability: n/a
• Course title: n/a

Background
The UBELT team in the English Language Centre at the University of Bath develop and deliver professional language testing for overseas medical professionals coming to work in the UK. Clients include the NHS, Tesco Pharmacy, Boots, Integrated Dental Holdings, Oasis Dental Care as-well-as various small independent medical employers. Kevin Renfrew, UBELT Business Development Manager, was looking for an online solution, available at any time and from any geographical location, which could be used as a diagnostic test for Boots. This would be used for pre-screening of pharmacy candidates, to enable identification of candidates who were ready to take the UBELT assessment.

What actually took place?
Kevin approached the e-learning team with a set of requirements and through discussion with Learning Technologist, Roger Gardner, identified the possibilities and limitations of the Moodle quiz tool. Although the tool did not meet all of his requirements, Kevin concluded that there were sufficient question types, reporting and other functionality.

Roger ran a customised workshop on use of the Quiz for Kevin and a number of his colleagues, after which they began developing question banks and setting up quizzes. The tests have 3 parts in Moodle:

1. Reading and Use of English Language. This makes use of gap fill, multiple choice and short answer questions.
2. Writing. This uses the Moodle essay question for submission but is marked manually
3. Listening. This involves audio clips and makes use of gap fill, multiple choice and short answer questions.

Time limits were set on the quizzes, and as the participants were not University of Bath students, Kevin contacted BUCS (Bath University Computing Services) to get temporary WAAA user accounts set up which were used to give access to the tests.

What was the role of the technology?
One of the requirements for the tests was that participants should not be disadvantaged by the fact that they were conducted online, for example by lack of familiarity with the technology. So the UBELT team set up practice tests which participants could access as many times as they liked in order to familiarise themselves with the environment, and identify any technical issues so these could be resolved before the real test, e.g. ensuring audio was working correctly.

The technology enabled participants to access the tests at a time and place convenient to themselves, and gave them an opportunity to practise assessments. The practice tests had formative feedback so that participants could see where they had gone wrong.

Outcomes
Were the objectives met?
Broadly, yes. Initial indications suggest that final results on the UBELT test show good concurrency with the online diagnostic, though participant numbers are too small to be conclusive. The online diagnostic has also provided positive additional practice for those attempting the full UBELT Test.

How did the participants find it?
The participants found the assessments useful and also that they helped to de-mystify the actual test event though these are not conducted online. Feedback from test commissioners (those who request and book the tests on behalf of the client or candidate) stated that the tests were easy to use, well laid out, effective and usable.

Issues / problems
Because of the question types employed UBELT has not been able to use the item analysis function and has had to rely on undertaking item analysis of questions manually to refine and improve their question designs.

On investigation by the e-learning team this was discovered to be caused by a wider Moodle bug, concerning item analysis of gap-fill questions, which is as yet unresolved.

UBELT would also like more flexibility in framing questions, for example getting audio to start automatically when a page is loaded rather than requiring the student to click to start it.

What tips do you have for others?
1. Ensure you have a clear objectives beforehand
2. Allow plenty of time for planning and developing quizzes, especially if working on them intermittently. Also allow time for housekeeping e.g. deleting duplicate questions.

Plans for the future
The team are considering broadening out their use of online diagnostic tests to all pharmacy candidates, other medical professions for which UBELT Tests exist, and are also investigating its use for pre-assessment of students in European medical schools.

• Name & email: Lindsay Jordan (l.jordan@bath.ac.uk)

• Keywords (pedadogical): online (distance learning) group work
• Keywords (technical): Discussion forum, chat, wiki, moodle

• Discipline: n/a
• Year / ability: n/a
• Course title: MSc International Construction Management

---

Background

Aims and objectives (what were you trying to do?)

We wanted to develop and pilot an assessment activity that enabled the students to develop and demonstrate negotiation skills relevant to the learning outcomes of the Construction Law module, such as conflict management and dispute resolution. We also wanted to give students the opportunity to apply their understanding of contract law to practice.

What actually took place (describe what you did)?

We took an existing dispute resolution role-play scenario that Jim had used previously with students at UWE, and adapted it to become an online activity for distance learners. Within the Moodle course for the module, we provided background information in a series of web pages, a group sign-up wiki, a negotiation forum for students to communicate privately within their groups, a group wiki for each group to produce an agenda for their negotiation meeting and record the outcomes of their meeting, and a Chat activity to give students the option of synchronous negotiation.

The task was clearly divided into three phases and four distinct deadlines were set, the first being the date by which students needed to have signed up to a working group in the sign-up wiki.

What was the role of the technology (what did the technology add?)?

The Forum, Wiki and Chat tools allowed students to easily select their own working groups, to share and comment on position statements within their own groups, to negotiate synchronously or asynchronously and to collaborate on producing an agenda and recording the outcomes of their negotiation, regardless of their geographical location.
The technology also made it easy for us to monitor student participation and group progress, and to contact groups or individuals where necessary throughout the task.

---

Outcomes

Were the objectives met?

The evidence gained so far supports the objectives having been met. 100% participation was achieved and all students received grades of 55% or more for this task, with many students achieving over 70%. Two students (out of 33) chose to defer their studies part-way through the task, but the design of the activity had allowed for this possibility and their fellow group members were not disadvantaged. All students went on to achieved C grades and above in the module assignment – this was a significant improvement on previous cohorts and on the grades achieved in previous modules by this cohort. Students will sit the module examination at the end of October 2008.

How did the students find it?

Although several students found some aspects of the task frustrating (for example, arranging to meet in the synchronous chat and one group member being absent) they accepted that similar problems occur when attempting to negotiate through offline means, and they thought the task was fun, engaging and a good opportunity to experience the dispute resolution process. Students will complete formal evaluations of the module and the negotiation task after their examination at the end of October.

Were there any unexpected outcomes?

A few students who hadn’t previously performed particularly well in written assignments and formal examinations appeared to excel in this activity. It will be interesting to compare their assignment and exam grades for this module with those of other modules, and to ask those students in particular how they felt the online role play task enhanced their learning on this module.

A secondary outcome that we could have reasonably expected was an improvement in the cohesion of the group, with new links and friendships being made between group members.

---

An outline of this case study has been submitted to CEBE, the Higher Education Acadmey’s Centre for Education in the Built Environment. The content of this submission can be viewed here.