The ORBIT Self-Assessment Tool

The self-assessment tool is one of the services provided by the ORBIT project. This editorial explains the background and principles of implementation and discusses the current state of development. It reviews possible strengths and weaknesses and charts the course of further development. ORBIT, self-assessment tool, RRI, responsible research and innovation Citation: Stahl, B. (2017). The ORBIT Self-Assessment Tool. ORBIT Journal, 1(2). https://doi.org/10.29297/orbit.v1i2.59 The ORBIT Journal: Current Status and Next Steps The ORBIT journal was officially launched in October 2017. I am delighted to announce that we have in place an editorial team consisting of four associate editors in addition to the two main editors of the journal. We furthermore have started to recruit members of the editorial board. For a current list of the entire editorial team please check https://www.orbit-rri.org/ojs/index.php/orbit/editorial-board. The process of recruiting editorial board members is ongoing and we continue to look for individuals who are interested in the area of RRI and who would like to contribute to the shaping of the journal and the wider debate. Setting up a new Journal is not a trivial matter and will require the continuous and ongoing contribution of the community of individuals interested in supporting it. At this point the journal is integrated into the ORBIT website. It is based on a recently published WordPress plug-in of the well-established Open Journal System (https://www.orbitrri.org/ojs). We have been provided with an ISSN (2515-8562) and can therefore be considered an academic journal that is functional and operational. This does not mean, however, that we have finished establishing the journal. The ORBIT journal clearly aims to be a respectable journal whose output will contribute to the academic debate and which is of sufficient standing to ensure that authors benefit from publication in terms of their personal and professional development. Achieving this is a process that will take time. We will need to demonstrate that there is a continuing and sustainable interest in the topics


ORBIT Journal Issue 1:2
Following this brief update of the development of the ORBIT journal, I would like to point to the papers that constitute the second issue.This issue contains the second batch of papers that were part of the 2017 ETHICOMP conference.These papers continue to reflect the breadth of issues that constitute the body of work in the field.As our recent review of the computing-oriented literature has shown (Stahl, Timmermans, & Mittelstadt, 2016), privacy and data protection remain the biggest topic in the area.This is reflected in the second issue of the ORBIT journal which carries several papers that look at privacy in social networks (Chutikulrungsee & Burmeister, 2017), brain-computer interfaces (Wahlstrom, Fairweather, & Ashman, 2017), cultural differences in data sensitivity (Fukuta, Murata, Adams, Orito, & Palma, 2017) and legal aspects of privacy in US law (Robison, 2017).
However, the second issue also demonstrates that ethical issues in computing go far beyond questions of privacy and data protection.Application areas of interest include simulated game worlds (Flick, Dennis, & Reinhard, 2017), decision making with algorithms (Persson & Kavathatzopoulos, 2017), wearable devices (Murata et al., 2017) and conversational agents (Miller, Wolf, & Grodzinsky, 2017).
A further stream of research and reflection refers to the way in which ethical issues can be integrated into research methods, such as netnography (Tuikka, Nguyen, & Kimppa, 2017) and, more prominently, which methods can be used to integrate ethics in ICT design and use, for example through value design engineering (Lilley & Moras, 2017), but also through well-established approaches such as codes of ethics (Brinkman et al., 2017).One interesting stream of activity is the use of alternative and non-traditional ways of exploring ethical issues (Jensen, 2017;Jensen & Vistisen, 2017).
Overall, this second issue shows the breadth of issues and ways of dealing with them that responsible research and innovation in ICT requires.This is a sign of the vitality of the field and the need for the ORBIT journal Having provided an update of the progress of the journal and the content of the second issue, I would like to use the remainder of this editorial to introduce and discuss another key component of the ORBIT services, namely the self-assessment tool.

The ORBIT Self-Assessment Tool
The self-assessment tool (https://www.orbit-rri.org/self-assessment/) is part of the suite of services that the ORBIT project offers to its users and stakeholders.In order to explain its principles and development, it is important to quickly outline the rationale for providing such a tool and its role in the overall RRI debate.This document will use a set of questions and answers to highlight key features of the tool.

Why do we need a self-assessment tool in ORBIT?
RRI requires the stakeholders involved in research and innovation to reflect on their practices, accept responsibilities and be responsive to one another with a view to ensuring that the purpose, process and product of research and innovation are aligned with legitimate societal expectations.This is a very tall order and the discourse around responsible research and innovation demonstrates that it will require significant conceptual clarification and empirical insights.Much of the work required for establishing a culture of RRI, which is part of what the ORBIT project is tasked to achieve, will require the development and sharing of good practice.This, in turn, is to a large degree a social exercise which means that mechanisms of communication and exchange have to be used that foster this communication and development of shared views.
However, there is also a legitimate place for individual reflection of current practices.The EPSRC has put forward the AREA acronym as a way of approaching RRI (https://www.epsrc.ac.uk/research/framework/).The R in this acronym stands for reflection.Reflection can take many forms and in most cases when we think about reflection is probably in terms of the process of thinking about a particular subject.And this will in many cases be exactly what RRI calls for.The problem with this approach to reflection is that it requires the individual or collective undertaking the reflection to have a significant body of knowledge to be able to reflect in a purposeful manner.In the case of RRI, which to the majority of the individual working in ICT is still relatively unknown concept, this is unlikely to be particularly helpful or lead to the desired outcomes.This means that the process of reflection in RRI is likely to benefit from a more clearly defined structure.And this is exactly what the ORBIT self-assessment tool is meant to provide.It is a tool which guides somebody interested in reflecting on a research or innovation activity through a set of steps with the aim of highlighting potential areas of relevance and thereby triggering ideas for actions that will help put RRI in practice.
In addition to this individual benefit of the ORBIT self-assessment tool, there is a second purpose that it is hoped to achieve.This is the development of a baseline that will allow the comparison of RRI between projects, users, technologies or sub-disciplines.The idea here is not so much that it would be possible to identify objective measures or metrics of RRI but rather that an attempt to quantify stages of RRI development will provide an input into discussions about areas worth developing further.

What is the basis of the self-assessment tool?
The idea of a self-assessment tool is by no means novel, not even in the context of RRI.There is a number of projects in the area of RRI that have developed either full-blown self-assessment tools or documents or other activities that serve a similar purpose.The first one was probably developed by the Karim project which went under the name of responsible innovation diagnosis in ICT (http://test4.net/cfi/).This project focused in particular on small and medium-sized enterprises active in the area of ICT.The probably most prominent example of such a tool is the RRI-tools self-reflection tool (https://www.rri-tools.eu/self-reflection-tool). A third example is the responsibility navigator that was a result of the ResAgorA project (http://responsibilitynavigator.eu/navigator/).
In addition to these examples of tools that are already available, there is a significant number of project outcomes, deliverables as well as publications which provide further insight into components of RRI that need to be reflected upon (Expert Group on Policy Indicators, 2015;Iatridis & Schroeder, 2015;Stahl, Flick, & Timmermans, 2017).
All of these existing tools and approaches have gaps or blind spots that make them difficult to simply use for the purposes required by the UK ICT community.Some of them are too generic and do not allow for a specific reflection of issues related to ICT.Others are broadly discursive and predominantly qualitative, which renders them difficult to use for some of the purposes described earlier, such as developing a quantitative baseline or providing automated feedback.A key weakness that several of them share is that they are linked to funded projects and the organisational and intellectual support ceased with the end of project funding.
The ORBIT self-assessment tool draws on all of these sources and rich discourse around RRI to provide the foundation of a system that can specifically address the needs of its user base.Its theoretical underpinning is the RRI maturity model (Stahl, Obach, et al., 2017) which explicitly set out to synthesise different perspectives on RRI with a view to providing an overarching perspective on the level of RRI maturity across projects or organisations.Crucially, it combines the EPSRC vision of RRI as expressed in the AREA framework with the highly influential view of RRI as proposed by the European commission that emphasises six different keys (gender, science education, open access, ethics, public engagement and governance).
Very briefly, the RRI maturity model, which is based on an existing discourse around maturity models in other areas, such as the capability maturity model (Paulk, 2002), postulates that there are components of RRI that can develop in terms of their maturity from unaware, where the concept is not even discussed, all the way to strategic, where RRI becomes a key aspect of a project or organisation.This idea is expressed in the following figure:

Strategic
Organisation is not aware of RRI or its components and does not incorporate it in its processes Organisation reacts to external pressures concerning aspects of RRI and experiments concerning appropriate processes Organisation has a definition of (components of) RRI and has integrated these into its business processes Organisation realises the benefits of RRI and seeks to proactively and increasingly integrate these into its business processes Organisation has adopted RRI as a component of its strategic framework and aims to ensure all R&D activities cover all (or most) RRI components Figure 1: Levels of RRI maturity, adopted from (Stahl, Obach, et al., 2017) This principle of the RRI maturity is then to be applied to the various components of RRI.One way of categorising these components is to use the AREA-4P version of RRI (Jirotka, Grimpe, Stahl, Hartswood, & Eden, 2017), with its focus on purpose (motivation), process and product (outcome) of research and innovation.These can then be broken down in more detail as shown below.
•  Representation of RRI maturity of three case studies, adopted from (Stahl, Obach, et al., 2017) The ORBIT RRI self-assessment tool is based on these ideas, but it goes further than the one originally suggested by offering more specific insight for particular issues related to ICT in general, application areas or technologies.

Who can use the tool and for which purposes?
The ORBIT self-assessment tool will be openly available on the ORBIT website and therefore be open to be used by anybody.However, there are certain types of users and interests which we assume are likely to represent the bulk of likely uses.The following three indicated use cases represent the ORBIT team's assumptions about such likely use.
1.A researcher (either publicly or privately funded) is working on a proposal for a publicly funded project.This proposal is linked to RRI requirements (e.g it is a H2020 proposal or a UK EPSRC proposal) and the proposer has to ensure that funding requirements are met.
2. A leader of an R&D department wants to find out whether the work undertaken in the department conforms to existing standards and reflects good practice.They want to persuade the organisation to invest into more R&D and therefore want to make sure its processes are optimised.
3. An organisation that claims to be socially responsible wants to ensure that this claim is reflected in all of its activities, including R&D.They are looking for a way to test this with regards to their R&D activities It is important to see that in principle the tool should be relevant for both individual researchers and organisations.It should cover both publicly and privately funded research and innovation activities.

What is the content of the self-assessment tool?
In practice they ORBIT self-assessment tool is implemented in the form of an online survey.In order to allow the identification of the level of achievement of RRI in a particular aspect by a particular user, the majority of the questions of the survey take the form of a five point Likert scale comment on a statement provided in the question, ranging from fully disagree to fully agree.
In order to achieve the aim of covering both the EPSRC and the EU views of RRI as well as more specific questions around ICT, it's technologies and applications, the ORBIT self-assessment tool is structured in the following sections: 1. Front End the starting point of the survey is a data use agreement where the user agrees to the use of the data they provide for the purposes of the self-assessment tool.This is predominantly the provision of services to the user, but it explicitly also covers the use of abstract and anonymous data for the purpose of doing research on RRI and establishing baselines.The user is then asked questions about their work and experience, the practices in their organisation and/or project.2. Awareness of RRI the first main section covers the awareness of RRI by the user and their organisation as well as the purpose and motivation for engaging with RRI and the selfassessment tool.

Basic EPSRC -AREA-4P
in the subsequent section the tool deals with the main components of RRI according to the EPSRC AREA framework, namely anticipation, reflection and engagement.

EU topics not yet covered
in order to ensure that the self-assessment tool is of value with regards to the EU framework of RRI, those topics that are not already covered, namely research governance, gender equality, open science, science education and research ethics are included.5. ICT general following the generic aspects of RRI, the tool then focuses on ICT specific issues, such as data protection, data storage, consequences for employment or machine autonomy.6. Specific applications or technologies depending on the responses that users provided in the first section of tool, specific items relating to applications or technologies will be asked.This section of the tool is yet to be developed, which requires drawing on the expertise of the various sub-communities working on particular applications or technologies, such as artificial intelligence, big data analytics, robotics, autonomous systems etc. 7. Feedback on the survey the final section will allow users to give feedback on the survey itself, identify whether it met their needs and highlight where it needs to be improved and developed.
For each of these components and subcomponents, a number of statements was developed, based on existing tools in the RRI literature that represent particular aspects of RRI and that allow assessing the level of maturity with regards to this particular component.
What is the output of the tool, which feedback do users receive?
Upon completion of the survey the user will be provided with an automatically generated overview of their RRI maturity, based on the score that was recorded in their answers.This initial feedback which will take the form of a radar diagram as indicated earlier will be supplemented with a set of links to further help and support that users can make use of, to ensure that they undertake their research or innovation activities responsibly.
In addition, the self-assessment tool will also point to the various other services that are provided by the ORBIT project, which covers most of the aspects of RRI.
Finally, the survey will encourage users to make contact with other members of the OR-BIT community, to foster the development of an RRI culture and the sharing of good practice.
What are the limits of the tool?
The purpose of the self-assessment tool is to provide a quick overview of the various components involved in RRI.It therefore and by necessity simplifies potentially complicated relationships and reduces them to quantitative scores.Such quantification of complex social relationships is always misleading and cannot capture the nuance of the underlying reality.In addition the scores that are produced by the ORBIT self-assessment survey are initially based exclusively on self-evaluation.This means that it is possible that the respondent did not fully understand the question or interpreted it in a way contrary to its original meaning.By relying on self-evaluation it is furthermore difficult to directly compare individual scores, as these will be based on the perceptions of the respondents.
These limitations of the tool need to be clear and understood, in order to ensure that the use of the tool and its output are beneficial rather than misleading.The value of engaging with the tool lies less in the answers that the individual user provides and more in the process of engaging with it.At the same time the quantification of the responses has the advantage that the user can compare his or her own initial scores with scores generated at a later point or with regards to different projects.It is furthermore reasonable to assume that once the tool has been used a significant amount of time, the average scores will represent a widespread view of particular items and the comparison of individual scores can at least give an indication of where a particular respondent or project is located with regards to the majority of comparable projects.
With this in mind it is important to underline that the scores generated by the selfassessment tool need context and detail and require critical reflection.This means that even a perfect score in the self-assessment tool does not guarantee that the researcher innovation activities undertaken in response manner and, vice versa, i.e. a low score does not necessarily imply lack of responsibility.

What is the status of the tool?
The self-assessment tool was developed during the first nine months of 2017.Initial rounds of review and feedback be undertaken during the summer of 2017 and it was implemented as an online survey on the ORBIT system from October 2017.We expect the tool to develop based on feedback from users.The online survey contains questions that allow users to provide feedback with regards to further development of the tool which we expect to be used and lead to further refinement of the ideas and principles.
In addition to this development based on user feedback, there will be specific areas which will be developed in more detail.In its first iteration the self-assessment tool will be identical for all users and focus on generic issues of RRI as well as ICT.It is intended to develop the tool to become more specific by integrating particular items and issues that are specific to particular applications or technologies.This will require input from the ICT community to firstly identify which areas are most in need of further reflection and secondly to then identify which issues they raise and how these can be represented in the self-assessment tool.
Overall we expect the tool to continue to develop and not to be used as a static instrument.There may be scope for a more interactive and user-friendly implementation beyond a simple online survey.Again, feedback from the ICT research community is welcome to stimulate the development of the tool.
For now we would like to encourage members of the community to try it and give feedback to help us make sure that the self-assessment tool fulfils its purpose and helps users to reflect on the level of RRI in their organisations and projects.
Funding:The work of the ORBIT project is funded by the EPSRC.In addition, the research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 609817 (Responsible-Industry).