Benchmarking Music Performance in Music Education Institutions

By Ron Brooker

With regard to the changing face of music education in Australia and the demands of servicing a global market, there is an increasing need for music training institutions to be transparent in their methods of establishing and verifying standards. How do Australian music schools determine the standards for progression and graduation at their institutions? Are they comparable? How do these standards match with those of the music profession? Has there been any benchmarking of music performance standards in Australian music education institutions.

These questions have been the subject of an ongoing study at the Sydney Conservatorium of Music where it had been observed that assessment panel members could allocate a wide variety of grades for the same performances. The means of establishing benchmarks for music performance were developed by ascertaining the processes used by various examiners, their expectations and their value systems, and their experience and training. The role of examiner training, the perceptions and beliefs of students, and the use of audio-visual recordings have all been canvassed in this attempt to establish an institution of "best practice".

The perceived need of developing this wide consensus on the standards at each level of a performer's career has been addressed locally. Student, teacher and examiner training, together with open and regular communication, are now being advanced as the model for all tertiary music training institutions.

Thank you for inviting me to speak at this conference.

While choosing the title for my paper, "Benchmarking Music Performance in Music Education Institutions", I was reflecting on how music education in Australia has changed in the many years I have worked at the Sydney Conservatorium. Recent events, like the return to the Macquarie Street site, the amalgamation with the University of Sydney Music Department, and the soon to be published history of the Sydney Conservatorium of Music, have highlighted to me that many of the practices of the past are no longer appropriate and now should be reviewed. As my abstract points out, music education institutions in Australia are no longer providers for only limited local communities. They now must respond to the demands of a global market, a market in which there is increasing competition for students, constant monitoring of institutions' budgetary management, and regular quality assurance reviews. Centrally linked with these changes is "an increasing need for music training institutions to be transparent in their methods of establishing and verifying standards". The basic question that I will be addressing in this paper is: "How do Australian music schools determine the standards for progression and graduation at their institutions?"

With the advantage of having listened to Nathan Waks and Ros Dunlop's papers yesterday, I have here also perceived that, at least in their respective views, these standards of student performance in Australian music schools have in recent years been progressively compromised. Furthermore, I have noted that the standards to which they referred were the ones of being acceptable to some very experienced, widely recognised teacher/performers, and of being quickly absorbed by the music profession. Standards such as these can be readily quantified in retrospect but they are ones that require examiners of the legendary quality of John Painter, Lois Simpson, and Winifred Durie, or perhaps a ten-year reunion of a particular graduating class. Unfortunately, the immediacy of providing grades for music performance students precludes these retrospective measures. Instead I have observed that different music schools have variously constituted panels of different examiners who award marks to students who either progress/graduate or fail. My question remains: "What are these standards that they are applying?" There are also the associated questions: "Are these standards comparable (between Schools, between Units, and even between instruments within Units)?" "How do these standards compare with those of the music profession?" "Has there been any benchmarking of music performance standards among Australian music schools?" My answer to all of these questions is "I don't know", but, in a manner similar to the quest for the Golden Fleece, my colleagues and I have started the search.

In the beginning, I looked at my own institution's Student Records to see what has happened and I found that students studying music performance were consistently being awarded higher marks than those recommended in the distribution quota by the University's Academic Board. See Figures 1 & 2.

These data require two comments. First, it is a moot point whether faculties at any university, including the University of Sydney, do adhere to the quota recommendation, even though I am aware of at least one Dean who has demanded it of her staff. Second, there exists a generally perceived attitude amongst students and teachers of the creative arts (and the arts community generally) that the award of a grade below a Credit is, in fact, a Fail. Support for this interpretation comes from, not only discussions I have had with AMEB examiners, but also a recent graduation ceremony the Sydney Conservatorium shared with the Sydney College of the Arts (SCA), the visual arts faculty of the University. Contrary to the traditional (Academic Board recommended) distribution of 10-16% of Honours degrees in a graduating cohort, SCA awarded 37% of its graduands Honours degrees. Not surprisingly, given previous speakers' comments on falling standards, the Sydney Conservatorium awarded only 16%. Further support was obtained in another study where my colleagues and I interviewed Conservatorium examiners who, when asked, "What does a mark of 80% at a Senior Recital represent?" replied in these terms:

80% is locally defined as… "above average in terms of the cohort", "has impressive technique, but has further to go in maturity and artistic development", "above 90% is required for a student to have a realistic chance for a solo career… signifies flair, charisma, and exceptional natural gifts".

Further insights into the comparability of assessments that arose from my review of the Conservatorium's Student Records were that the marks awarded a particular student often varied across the years of a course (First Year was usually the highest), and that average marks awarded to all students in a particular Unit would differ from the average marks of another Unit. This variability, which assessment and evaluation theory tells us, reflects on the reliability (ie. consistency) and thus validity of the measurement, lead me to look at the panel process itself. The following paragraphs summarise this research. (OHTs 1- 6 used in the presentation are appended to this paper.)

Data has been collected from three recital examination periods: November, 1998, June, 1999, and November, 2000. Proformas that requested each panellist to record their independently determined marks for a performance and the performer‘s Principal Study Unit were distributed prior to each recital. After the examination, the Panel Chairs collected and returned the forms to the researcher. On the later data collectionoccasions, these proforma were amended to include the candidate’s name, the examiner’s Internal or External status, and most recently the examiner’s name. Data has been collected from 319 panels, 88 in 1988, 56 in 1999, and 175 in 2000. The panels examined 35 Brass, 29 Jazz, 48 Keyboard, 12 Percussion, 72 Strings, 36 Voice, and 87 Woodwind recitals.

The results showed the widest ranges (range= difference between highest and lowest marks awarded by a single panel) of independent marks awarded by individual examiners for each of the data collection periods were 43, 31, and 26 marks, respectively. The average range for all 319 panels was 9.42 marks (10.35 in 1998, 10.29 in 1999, and 8.66 in 2000). The median (middle) range of all panels was eight (8).

There were 68 panels (21%) who differed by less than five marks and, while 180 (56%) differed by less than ten, there were 139 panels (44%) who differed by more than the range of marks between Credit, Distinction, & HD. 62 panels (20%) had differences between examiners exceeding 15 marks.

As shown in Figure 3, the average ranges of panels indicated that there was also substantial variability in all panels. Brass and Woodwind panels were those with the largest internal variability.

These results have demonstrated that, at the Sydney Conservatorium at least, individual music performance examiners' assessments (by themselves) are not reliable, and that they are apparently applying different yardsticks, different criteria, and different weightings. Such unreliability questions the very validity of these music performance assessments.

What then do these examiners say about these findings? What are the standards (yardsticks) they use to guide them in determining grades? Twelve experienced examiners from all Units were asked their opinions about setting standards. In summary, they agreed that inconsistent standards did, in fact, exist between Units, and that the recital assessment process favoured certain (namely, the solo) instruments. Significantly, they identified experience as their main basis for setting standards. In particular, they listed: "personal experience" in listening to recitals at all levels, in performing, and in knowing the standards reached by past students; "accumulated experience" in knowing the comparative standards of the current cohort of student peers and students at other institutions around Australia and overseas; and "professional experience" in knowing the standards of performers in the music profession. Mixed together these experiences developed a personal confidence about their abilities to assess music performance in their Unit. But what do the students think?

To find out, I surveyed the current student body. (OHTs 7- 12 used in the presentation are found in Appendix 2.) Of the 139 surveyed, which represented 58% of the cohort, seventy-eight were Performance Majors and 61 Performance Minors. Because of time, I will here report only the findings for the Performance Majors.

The survey instrument comprised three sections: questions seeking demographic data on Gender, Instrumental Unit, Semester, Tuition History, and Grade Last Received in Performance Assessment; twelve ‘Assessment Issues’ statements phrased in the 1st person with a 5-point Likert scale (SA to SD); and space for ‘Any further comments?’

Data was obtained from the following Units: Brass (9); Keyboard (7); Strings (18); Voice (15); Woodwind (26);Percussion (2); Jazz (1). Most definite trends evident were on the following items:

assessments too frequent (1%:68% disagreeing),

the venues are suitable (28%:44% disagreeing),

higher standards in some Units (65% agreeing: 3%), and

factors other than musical affect assessments (15%:43% disagreeing).

Mixed views about the present arrangements were indicated for items:

more flexibility in timing of assessments (30% agreeing:20% disagreeing), and

performance assessments at the Conservatorium are fair (50%agreeing:20%).

Significant interactions were found between: Units on frequency of assessment; knowledge of a Unit's criteria; and the policy of having a moderator; a student's semester enrolment on frequency of assessment; and a student's Last Grade on the policy of independent marking. The findings for the Performance Minors were similar, but they also reported that they were being treated as 2^nd Class citizens. Some even claimed discrimination. Overall, the study has shown there is currently substantial disaffection about music performance assessment amongst the students.

So, what should happen? What is happening now?

Well, from the information so far obtained, I personally believe that all music performance courses should include assessment training wherein students are given the opportunity and the instruction on how to grade performances. Peer assessment with frequent staff review and substantial discussion of the process' underlying values and attitudes. Equally regular opportunities for examiners and teachers to communicate their views and their methods in performance assessment. Because of the ephemeral nature of live music performance, there are limitations on the use of recordings for assessment purposes, but we live in a time where critics regularly assess film, video, and DVD performances with conviction. I don't recommend that a recording of a live recital should replace the live panel assessment process, but I do hold that the availability of high quality recordings should enable and facilitate the training of examiners, and possibly the review of marks. Clearly, they should most definitely be used for teaching purposes.

What's happening now? Presently, there is a Peer Assessment program running at the University of Ulster, Belfast, which could be modelled as a course of "best practice". It is described in a paper by Hunter and Russ (1996). There is also a preliminary attempt at linking the development of performance examiners with DVD protocols and staff discussion at two large music training institutions. My firmly held belief is that "the knowledge" (the results of years of experience) of "the inner circle" (those with the power to examine) should be shared. Open and direct communication between all the stakeholders is the answer. Thank you.

Reference: Hunter, D. and Russ, M. (1996) Peer assessment in performance studies. British Journal of Music Education, 13, 67-78.

Appendix 1: OHTs 1-6

 

Appendix 2: OHTs 7-12