9.5 Seminal Sources - Research of Merit

This note is about what to look for when digging deeper, understanding and identifying what might be worthy research or material to include in a class. There are the obvious ways to find things; citations, references in the common text books. There are the big names at conferences and such.

What are other indicators of noteworthy impact? The following list was put together over many years, considering what signals can be found in submitted promotion files. Depending on the institution, a common requirement for moving from Associate to Full is 'high achievement', national and international recognition, noteworthy impact. The same list can help you ferret out noteworthy results and sources.

The more someone has on the list, the more of a Grand Master they might be, worthy of being listened to, sharing with your class. The list can also be used by those trying to develop a career (choices to make along the way) and what to potentially include in a promotion and tenure package.

Many of the indicators have a subtext as well; the number of replications (number in each category), who is worked with, the institution where the colleague is, where the researcher is interacting, and concentration of impact in a specific time window. Basically, these are types of indicators that might provide evidence that others know about the researcher’s work and consider it worthy and above the average.

There are quite a few possible indicators. The following list was updated in 2020. Some are direct indicators, some are proxies.

Worthy?

Here are some of the signals or cues that we often suggest that can be looked for when considering if someone's work is worth digging into, including in a course. If someone has a big, potentially worthy reputation.

• External PhD reviewer – national - institution, reputation of supervisor, what student went on to do
• External PhD reviewer – international - same as above - recognition as being one of the world's experts
• Use of original material in graduate, undergraduate courses at other institutions – where, duration, instructor using the material adds to the credibility, recognition
• Use of original material on assessments/tests at other institutions – where, instructor using the material, adding to impact, recognition of the original work
• Use of original material in other researcher’s research – level (MASc/PhD), where, when, supervisor, citations of leveraged work – not just citations, but actual incorporation, extending, replication
• Use of original material in applied areas (if applicable) – successful use of non-trivial academic results in the real world - research labs, research groups
• Use of original material in applied areas (if applicable) – successful use of non-trivial academic results in the real world, not just research labs, research groups, but in live use
• Adjunct positions – students worked and research at other institutions – level of work/collaboration – members of committees, co-supervision of students – when held, duration, colleagues
• Visiting scholar/sabbaticals – where, type (e.g., prestige, activity while on sabbatical - pure research vs ‘teaching’ a course), duration, with who, results from sabbatical
• Editor, Area Editor – journals

• Program/session chair, organizer – conferences, workshops – local, national, international
• Pioneer/innovator – creating workshop series, ‘parent/pioneer’ of field, first indicates higher impact
• Pioneer/innovator – creating workshop series, one of the early pioneers if not the first
• Types of citations/references in literature reviews (good journals) – mention, position as seminal work, number of references, impact --> indicators of body of work, depth and breadth suggested by ideas and topics contributed vs one concept and then reworked 30 times with minor incremental improvements in results
• Author/co-author of texts/monographs – original work – research oriented – beyond short papers
• Author/co-author of texts/monographs – original work – body of work interpreted for applied use, practitioners (especially in area like Engineering which is an applied science)
• Author/co-author of texts/monographs – overview of field, methodology, results – not own work, but showing world class expertise knowing the field, ability to interpret it
• Author/co-author of ‘teaching’ texts for field – possibly not own work, but showing expertise in the field and being able to interpret, explain it – recognized as a contribution to learning
• Author/co-author of texts/monographs – original work – body of work interpreted for applied use, practitioners (especially in area like Engineering which is an applied science)
• Editor/co-editor of handbooks, special issues – special topics, the field
• Handbook chapters – original work, reviews - prestige of handbook series is important
• Handbook chapters – reviews of the field, existing research - prestige of handbook series
• How introduced in handbook intro's (e.g., creator of x, early researcher on y)
• Book chapters – original work
• Book chapters – reviews
• Invited papers to special issues
• Invited keynotes – where, impact/prestige

• Co-authors – reputation, prestige, repeat collaboration with leading scholars
• Type and breadth of contributions – domain, methodology, enhancements and incremental vs foundational, exploratory, preliminary
• Grants, awards, chairs – research funding – consistent, above average awards
• Awards, recognition by societies, conferences, groups for contributions – body of work, young research awards
• Awards by societies, conferences for best papers
• Created field level ‘challenge benchmark’ – promoted by others as state-of-the-art challenge
• Referee service – journals, conferences – which ones, how often, timing/frequency
• Notes, references to researcher’s research methodology, concepts, results in government or similar reviews and reports – not in a contract report by author, but general field reports noting significant outcomes
• Contribution in national, international societies - e.g., National Chair
• Significant funding for labs, research activities - creating centers of recognized expertise
• Used as a referral point or 'expert' to experts - e.g., top professors at top institutions referring queries or research issues - the experts' expert
• Specific references in other researcher's publications, theses - as an inspiration, starting point for their work
• Possible external, international recognition, use by top think tanks as subject matter expert, senior advisor on topic, methods, results/interpretation
• Was the average ‘clock’ beat – e.g., getting to assoc by one year earlier? By more?
• Types of visitors that come to see the professor – e.g., international, big names visit and do research, give talks, co-supervise
• Is the level of work in say 5 years what others do in 6 or 7 – e.g., if this was at the normal ‘time’ of 6 years going up, and the candidate has done what was expected in 6 in 4-5 years, the candidate might be above average

Further reading

• Haddow, G. and B. Hammarfelt (2018). Quality, impact and quantification: Indicators and metrics use by social scientists. Journal of the Association for Information Science and Technology. 70, 1, pp. 16-26.
• Elliott, N., Begley, C. and Kleinpell, R. and A. Higgins (2014), The development of leadership outcome-indicators evaluating the contribution of clinical specialists and advanced practitioners to health care: a secondary analysis. Journal of advanced nursing, 70, 5, pp. 1078-1093.