On 23 October 2007, Fenja Theden of the Australian Academy of Science published the following article, which was also printed in Campus Review on 23 October 2007.
The forthcoming Research Quality Framework (RQF) process has been a topic of heated debate within most research organisations across Australia. A key part of the metrics vital to the process is the ranking of research outlets, mainly in the form of journals. As the time nears where draft ranked journal lists are released for broad sector feedback, the journal ranking exercise is likely to take centre stage within discussions surrounding the RQF.
In the future, the ranked journals will be used to help the RQF Assessment Panels consider a research group's relevant research outputs. The relative emphasis given to these metrics in the assessment will be determined by those panels. The current RQF journal ranking exercise therefore requires careful consideration from Australian researchers.
Because the RQF metrics are being developed on a discipline-specific basis, a large focus has been the importance of each research sector 'owning' its final journal rankings. That is to be achieved through each discipline determining and ranking its own journal lists.
To gain input from all disciplines within Australia, the Department of Education, Science and Training (DEST) has requested discipline-specific expert bodies undertake the task of assessing the quality of each research outlet in which Australian researchers publish.
So behind the scenes, a number of Australia's top expert bodies are systematically working their way through lists of hundreds of research outlets. These include mainly journals, but for some disciplines include conferences, book publishers and performance venues as well.
The Research Evaluation and Policy Project (REPP) at the Australian National University has been contracted by DEST to provide technical support for the development of RQF metrics. This includes the provision of sub-discipline specific journal lists as a starting point for those undertaking the initial journal ranking. The journal lists have been drawn from several sources and are exhaustive. By collecting research outlet data from Australian universities, REPP has been able to determine which journals are assigned to which discipline. Multiple listing is possible for some journals that focus on more than one sub-discipline.
DEST has engaged the National Academies Forum, consisting of Australia's four Learned Academies, to coordinate most of the expert bodies. Each Learned Academy has approached the task through its relevant committees and panels. For example, the Academy of Science has engaged its twenty National Committees, each highly specialised, to undertake the draft journal ranking. DEST has also approached a number of discipline-specific organisations to undertake rankings independently of the Academies.
Until now, the process has occurred largely within these small groups of Australia's expert bodies.
This is soon to change. As the draft journal lists are assembled, the Academies and other organisations involved in the draft ranking are preparing to publicise and widely distribute the lists for thorough sector feedback. Sector feedback is scheduled to occur in November this year, and will be coordinated by the Academies and other relevant professional organisations.
Throughout the RQF journal ranking exercise, independent overarching committees will be carefully monitoring the process. The Academy of Science, for example, has appointed a committee of senior Academy Fellows to review the ranking process of those disciplines in which the Academy has primary expertise. In addition to its monitoring role, the committee will adjudicate on any questions regarding the ranking of particular journals.
The journal lists themselves are based on the Research Fields, Courses and Disciplines (RFCD) classifications as published by the Australian Bureau of Statistics. These codes include recognised academic disciplines and related major sub-fields. The present review of the RFCD codes is still ongoing, but for at least the first year of the RQF exercise, the journal lists will be based on the current codes.
Wherever possible, REPP has attempted to construct journal lists at the 4-digit RFCD code level. Most science disciplines also have a 'general' journal list classified only to the 2-digit RFCD level, carrying articles from across all its sub-disciplines. Although journals have been assigned to only one or two discipline lists for the purpose of this exercise, researchers will have all their research outlets considered in the RQF assessment, not only those journals in the list that most closely matches their discipline.
To assist with the journal ranking task, useful data have been added to each list. These include existing citation data such as the European Reference Index for the Humanities (ERIH) factor for humanities research outlets.
The main determinant of the ranking, however, is the quality of the articles that each research outlet contains. Journals are being assigned to four prestige bands – A*, A, B, C – depending on their assessed current quality. Only those journals containing the highest quality papers from the world's leading researchers, the top 5% of journals, will be rated as 'A*'.
Journals ranked 'A' belong to the next 15%, and represents those journals also publishing very high quality papers with a significant proportion coming from the world's leading researchers. This group includes the leading journals in a number of sub-disciplines. The next 30% of journals are rated 'B', and the final 50% of peer reviewed journals are rated 'C'.
The first assessment process for the RQF will take place in 2008, with funding to flow from 2009. Independent of the RQF outcomes, the research outlet lists will be reviewed and revised where necessary on an annual basis.
© 2024 Australian Academy of Science
