The discussion about quality and assessing quality parameters in research is pertinent in many contexts. First of all, this involves assessing applications, i.e. rating scales as an instrument to assess the quality of the research. The other aspect is whether and how rating scales can be used to compare quality within or across fields of research. Comparing the quality of the fields as a part of their efforts to sponsor the best research and to be able to conduct horizontal prioritisation has a value per se for research funders. The third reason to use well-developed assessment scales is communication with the rest of the world on research and research funding. This paper is a summary of a seminar that discussed the use of rating scales in peer review assessment of research applications. The seminar was organised by the Swedish Research Council’s Scientific Council for Medicine and Health. Its purpose was to contribute to the ongoing discussion on quality assurance of the Swedish Research Council’s assessment processes. The seminar participants are solely responsible for the statements and conclusions put forward in the different sessions. The sessions thus do not necessarily represent the views of the Swedish Research Council.