• Name and e-mail: Kevin Renfrew (K.C.J.Renfrew@bath.ac.uk), Roger Gardner (r.gardner@bath.ac.uk)
  • Keywords (pedagogical): e-assessment, diagnostic, language testing
  • Keywords (technical):
  • Discipline: n/a
  • Year / ability: n/a
  • Course title: n/a

Background
The UBELT team in the English Language Centre at the University of Bath develop and deliver professional language testing for overseas medical professionals coming to work in the UK. Clients include the NHS, Tesco Pharmacy, Boots, Integrated Dental Holdings, Oasis Dental Care as-well-as various small independent medical employers. Kevin Renfrew, UBELT Business Development Manager, was looking for an online solution, available at any time and from any geographical location, which could be used as a diagnostic test for Boots. This would be used for pre-screening of pharmacy candidates, to enable identification of candidates who were ready to take the UBELT assessment.

What actually took place?
Kevin approached the e-learning team with a set of requirements and through discussion with Learning Technologist, Roger Gardner, identified the possibilities and limitations of the Moodle quiz tool. Although the tool did not meet all of his requirements, Kevin concluded that there were sufficient question types, reporting and other functionality.

Roger ran a customised workshop on use of the Quiz for Kevin and a number of his colleagues, after which they began developing question banks and setting up quizzes. The tests have 3 parts in Moodle:

1. Reading and Use of English Language. This makes use of gap fill, multiple choice and short answer questions.
2. Writing. This uses the Moodle essay question for submission but is marked manually
3. Listening. This involves audio clips and makes use of gap fill, multiple choice and short answer questions.

Time limits were set on the quizzes, and as the participants were not University of Bath students, Kevin contacted BUCS (Bath University Computing Services) to get temporary WAAA user accounts set up which were used to give access to the tests.

What was the role of the technology?
One of the requirements for the tests was that participants should not be disadvantaged by the fact that they were conducted online, for example by lack of familiarity with the technology. So the UBELT team set up practice tests which participants could access as many times as they liked in order to familiarise themselves with the environment, and identify any technical issues so these could be resolved before the real test, e.g. ensuring audio was working correctly.

The technology enabled participants to access the tests at a time and place convenient to themselves, and gave them an opportunity to practise assessments. The practice tests had formative feedback so that participants could see where they had gone wrong.

Outcomes
Were the objectives met?
Broadly, yes. Initial indications suggest that final results on the UBELT test show good concurrency with the online diagnostic, though participant numbers are too small to be conclusive. The online diagnostic has also provided positive additional practice for those attempting the full UBELT Test.

How did the participants find it?
The participants found the assessments useful and also that they helped to de-mystify the actual test event though these are not conducted online. Feedback from test commissioners (those who request and book the tests on behalf of the client or candidate) stated that the tests were easy to use, well laid out, effective and usable.

Issues / problems
Because of the question types employed UBELT has not been able to use the item analysis function and has had to rely on undertaking item analysis of questions manually to refine and improve their question designs.

On investigation by the e-learning team this was discovered to be caused by a wider Moodle bug, concerning item analysis of gap-fill questions, which is as yet unresolved.

UBELT would also like more flexibility in framing questions, for example getting audio to start automatically when a page is loaded rather than requiring the student to click to start it.

What tips do you have for others?
1. Ensure you have a clear objectives beforehand
2. Allow plenty of time for planning and developing quizzes, especially if working on them intermittently. Also allow time for housekeeping e.g. deleting duplicate questions.

Plans for the future
The team are considering broadening out their use of online diagnostic tests to all pharmacy candidates, other medical professions for which UBELT Tests exist, and are also investigating its use for pre-assessment of students in European medical schools.

Tags: , , , ,