Practicing responsible research and innovation in a crowdsourcing project in Norway

: The paper discusses the operationalization of responsible research and innovation (RRI), drawing on empirical materials from a research project that explored ICT-enabled tools and methods for crowdsourcing in urban environmental research and decision-making. An integrated model for RRI is developed from prior studies, in which socially responsible crowdsourcing is described as an iterative and recursive process of inclusion, anticipation, reflexivity and responsiveness on the purpose, process, product and people components of the crowdsourcing project. The paper outlines four important aspects that influence the practice of RRI: time,


Introduction
Crowdsourcing is an internet-based problem-solving and production model that leverages the collective intelligence of a distributed online community (Howe, 2006).With respect to public governance, Brabham (2015) defined crowdsourcing as a "natural extension of democratic engagement and citizen participation, but taken online with new tools" (p.12).For research, crowdsourcing can be understood as a way of collecting data and generating ideas through citizen science and public participation projects (Louv & Fitzpatrick, 2012).The term has been applied at multiple levels: from an open call for participation in online surveys (Behrend et al., 2011) to the creation of long-lasting online communities (Brabham, 2012), with a focus on relevant value propositions for different groups of participants (Barrett et al., 2016;Fossum et al., 2018).
Crowdsourcing can be helpful in various ways, for example, for data collection, the co-creation of ideas and solutions and the performance of specific tasks by online contributors (the crowd).However, it also raises a number of ethical and societal questions, including those surrounding privacy, data security, acceptability and desirability.One way to deal with such issues is to apply crowdsourcing within a responsible research and innovation (RRI) framework.This paper adopts Sutcliffe's (2011) broad definitions of research and innovation, where research refers to "systematic investigation to establish facts and reach new conclusions" and innovation refers to "the effective commercialization of a creation resulting from study and experimentation".However, there is undoubtedly increased overlap between research and innovation and it can be difficult to view one independently of the other.In this paper, the two terms are jointly used and referred to as R&I.
The concept of responsibility is not new to researchers and innovators.There have been calls for responsible conduct in science for centuries.However, what such responsibility actually entails has changed over time (Stilgoe et al., 2013).At the start of the 20th century, the dominant view, as put forward by Vannevar Bush (1945), was that the purpose of science is to deliver new and useful scientific knowledge to society, but to achieve this, science should be granted significant freedom.Current calls for responsibility can be said to have their origin in the perceived urgency for sustainability, in the light of various environmental and climate crises (Klaassen et al., 2017).Over the last century, a new perception emerged that scientific and technical solutions more often than not shifted consequences to other areas.A case in point would be CFC gases, which solved the problem of flammability and toxicity of the other coolants but turned out to have disastrous effects on the ozone layer (Gee et al., 2001).
The ambition of RRI is thus to increase the acceptability, desirability and sustainability of R&I activities (von Schomberg, 2011).RRI can also be viewed as a further development of ELSA (ethical, legal and societal aspects) studies, but with a stronger emphasis on the socio-economic effects, including innovation, job creation and competitiveness (Zwart et al., 2014).Both ELSA and RRI have been associated with emerging and disruptive technologies that could have a significant impact on society.As a result, the further employment and refinement of the RRI concept has so far been dominated by nanotechnologythe archetypal emerging and disruptive technology (Rip, 2014;Shelley-Egan et al. 2017).However, this does not imply that the relevance of RRI is limited to nanotechnology: it is increasingly important for most emerging technologies, not least ICT (Stahl et al., 2013).RRI enables the foregrounding of the potential effects of R&I on the environment and society before and during projects.RRI elicits potential new possibilities and requirements in research practice and management.Publicly funded research is typically conducted in projects, and the factors affecting the success of a project have been a topic of interest at least since the early 1960s (Pinto & Slevin, 1989).A project, according to the Project Management Institute (PMI) 1 , is as a temporary endeavor undertaken to produce a product, service or result; it should have definite starting and ending points (time), a budget (cost) and a clearly defined scope or magnitude of the work to be done, including specific performance requirements (quality).Thus, project management success is typically measured by the management of the triple constraints: the cost, quality/objective and time.It has, however, also been noted that project success and project management success are not one and the same (Nagesh & Thomas, 2015).Overrun of cost and time with a high-quality product may suggest that the project failed on project management but may nevertheless be considered a success, as this also eventually depends on its outcome and impact, including the satisfaction of stakeholders and potential customers.One could argue that RRI goes beyond the typical success measures in the project management literature, i.e. on time, in-budget, meeting the scope of requirement and quality, to encompass also an alignment of research and innovation with society's values.This would entail societal and ethical considerations, which would be a considerable expansion of the traditional interpretation of project management, particularly in the context of R&I.
The RRI concept has its origin in publicly funded research (Stahl, 2018).Since RRI became prominent, arguably in 2012 (Rip, 2014), it has been used by different funding agencies (RCN, 2017) and in several research projects.For example, it has been used for promoting multi-stakeholder engagement in environmental projects (cf. Ferri et al 2018).RRI is of course also relevant and important in privately funded projects and innovation processes.
This paper investigates how RRI can be operationalized in a research project on crowd-sourcing and specifically explores field-level challenges and possibilities in implementing and practicing RRI.While there are already studies on RRI in the domain of the information communication technologies (ICTs) (cf. Ferri et al., 2018;Yaghmaei, 2018), few have detailed how the RRI concept is applied in an actual practical project setting.Thus, this study responds to the recent call by Burget et al. (2017) for more empirical studies that substantially elaborate the RRI concept in practice.Drawing on available RRI frameworks and studying RRI at the project level, the paper aspires to contribute conceptually and empirically to the recently consolidated but still emerging literature on RRI.The findings and reflections here intend to contribute in the further development of the RRI framework and thereby also its wider practical relevance, as well as to current academic discussions of the future of RRI (e.g.Klaassen et al., 2017;Asveld & van Dam-Mieras, 2017;Kuzma & Roberts, 2018;Fisher, 2018).
The remainder of the paper is organized as follows.In the following section the concept of RRI is introduced through a focus on its components and dimensions.This is the basis for establishing an RRI model used as an analytical framework in this paper.Next, the methodology used for data gathering and analysis is described.In section three, the research project under study and the cases that make up the empirical material are presented.Section four presents, analyses and discusses the empirical case material using the RRI model.The paper ends with some concluding remarks.

Responsible Research and Innovation
In a widely used definition, RRI is described as: "(…) a transparent, interactive process by which societal actors and innovators become mutually responsive to each other with a view on ethical acceptability, sustainability and societal desirability of the innovation process and its marketable products (in order to allow a proper embedding of scientific and technological advances) in our society" (von Schomberg, 2011, p. 9).
Using the core concept of responsibility in RRI as a point of departure, Stahl (2013) suggested that RRI could be further refined as "a higher-level responsibility or meta-responsibility that aims to shape, maintain, develop, coordinate and align existing and novel research and innovation related processes, actors and responsibilities with a view to ensuring desirable and acceptable research outcomes" (p.708).Stahl (2013) also views RRI as a space defined by activities, actors and norms, thus emphasizing the need for a systemic approach to determine the acceptability and desirability of R&I.
For research, RRI responds to the observations of Latour (1998):"there was no direct connection between scientific results and the larger context of society, which could do more than slow down and speed up the advancement of an autonomous science" (p.208).Connected to innovations in ICT, the introduction of RRI may be viewed as a reinforcement and extension of existing models and conceptual frameworks, such as user acceptance models (Davis, 1989) and participatory action research approaches (Baskerville, 1999).While these models emphasize user involvement for increasing the acceptance of technology products and services, Scandinavian research on information system development stresses workplace democracy as a driver for involving users and giving them power over design decisions (Bjerknes & Bratteteig, 1995).Yet, the discussions around user involvement and participation have typically been limited to the organizational level.Participatory approaches have also been used mainly to promote the traditional view of using participation for "right impact" and risk mitigation (Owen et al., 2012).The concept of engagement in RRI expands the discourse of user engagement to the societal level for critically investigating the social, economic and ethical dilemmas and opportunities surrounding the decisions, actions and outcomes related to R&I.RRI is still an emerging concept.However, it is more than a theoretical debate.It has become an overarching practical R&I governance approach for ensuring the desirability, acceptability and sustainability of ICT-based innovations (Jirotka et al., 2017, Stahl et al., 2017).In the next section, we present the components and dimensions of RRI, the aim being to combine these in an integrated model.

Components
RRI is often claimed to be distinguished by four major components related to R&I: purpose, process, product and people (Jirotka et al., 2017;Owen et al., 2012;Klaassen et al., 2017).
The purpose relates to the motivations for the R&I.Here, RRI practitioners deliberate on questions such as the following.Why is this effort undertaken?What stakeholders may benefit, and which stakeholders may end up with (increased) risks?This goes beyond thinking what we would not like science to deliver, but actively reflecting on what it should deliver (Owen et al., 2012),thereby also enabling anticipatory governance (Burget et al., 2017).Under an RRI framework, there should be inclusive deliberation on the direction of research and innovation from the outset of a project.This should involve relevant stakeholders that are directly and indirectly impacted by the outcome of the effort, including the public.Nonetheless, determining who constitutes the public in a specific context and how the biases of social norms and values should be accounted for in the production of knowledge remains a challenge (Glerup & Horst, 2014).
Process is the means by which research and innovation are actually performed, and it takes place following the mechanisms of reflection, anticipation and inclusive deliberation (Owen et al., 2012).It is a call for public engagement in R&I processes.Recently, approaches such as open innovation, citizen juries and crowdsourcing have been tested as part of the R&I processes for increasing public engagement, value creation and impact (Lövbrand et al., 2011;Carayannis & Rakhmatullin, 2014;Brabham, 2015).Public engagement is also one of the key elements of RRI policy, as discussed by the Expert Group on Policy Indicators for Responsible Research and Innovation (Strand et al., 2015).As is well known in the project management field, time is an important factor in the R&I process.It may also influence the level of inclusiveness.Practicing RRI potentially presupposes longer project time.The presence or absence of supporting structural elements, including legislative prerequisites (e.g.gender quotas) and established peer review practices for scientific publications (Forsberg et al., 2018), is pertinent to discussions on RRI processes.
The product component deals with the anticipation of potential uses of research outcomes (Jirotka et al., 2017) and the inscription of values in research and innovation products (Simon, 2017).Such values include economy, fairness, sustainability and privacy.To what extent specific values are emphasized depends on the various groups of societal actors who further shape the outcome.In line with this approach, Stahl (2013) recognizes multiple responsibilities that come from various actors' professional roles, norms and legal frameworks (RRI Tools, website, n.d.).This implies the need to negotiate responsibilities and identify those that are dominant and hence demand particular attention.
The presence of multiple actors with various responsibilities brings to prominence the fourth component of RRI: people.Some scholars, e.g.Mumford (2000), argue that, in managing creativity and innovation, people are central and that effective practices must consider the individual, the group, the organization as a whole and the strategic context in which an organization operates.The people component is also about who should be involved and whether the needs of all relevant stakeholder groups are considered in the current R&I approach of a project.RRI requires the involved people to represent interdisciplinarity and trans-organizational approaches (Jirotka et al., 2017).

Dimensions
Using von Schomberg's (2011) definition as a point of departure, Stilgoe et al. (2013) suggested an RRI framework that comprises four process dimensions: inclusion, anticipation, reflexivity and responsiveness, and related approaches.These dimensions are further described below.We also connect them to the components of RRI, thus establishing the conceptual model that guides the Discussion section of this paper.

Inclusion
Inclusion covers activities that involve the identification of stakeholders who are directly and indirectly affected by the R&I process.This dimension acknowledges the presence of different kinds of knowledge, including that of citizens, and it calls for their participation in the design and goal-setting dimensions of a project (Jirotka et al., 2017).So, a wide range of actors and publics should be involved in the entire R&I process, from the start through to the end.This includes their involvement in deliberation and decisionmaking as a way to create scientific knowledge that is of a higher quality, thanks to the input of a broader range of expertise, disciplines and perspectives.Inclusion clearly relates to the people component and responds to the questions: Who is or should be a part of the process?Who has a stake in the project's outcome?How diverse is the stakeholder group?Importantly, this often entails making the R&I process more transparent.
Once relevant stakeholders have been identified, R&I projects should create a space for stakeholder dialogue so that the possible risks, unintended social effects and potential benefits of the ICT-based activities, solutions and innovations can be anticipated.This process implies gaining an understanding of the values, needs and wishes of society.Arguably, mixed approaches for inclusion tailored towards the stakeholders (publics) of a project are beneficial for participation.For instance, the Norwegian Assisted Living project aimed at responsible innovations for dignified living for persons with mild cognitive impairment or dementia used dialogue cafes, technology trials, surveys and focus groups to explore the challenges in the daily lives of the elderly and to recommend appropriate technological solutions (Forsberg & Thorstensen, 2018).They did, however, experience some challenges in ascertaining the actual needs of the elderly, because the dialogue café was designed with a focus on general user stories in order to respect privacy by minimizing the revelation of personal information.In such cases, other inclusion approaches could be necessary.Magoni et al. (2018), for example, found one-to-one meetings as the most promising strategy for engaging industry actors; while scientists were more easily reached through classic tools, such as scientific debate in meetings, conferences and papers.Thus, different approaches have to be used depending on the project's purpose and context.When arranging for stakeholder inclusion and dialogues, it is also important to be aware of potential conflicting value frames.This would require openness to criticism and changes in direction (Blok, 2014) in the formulation of the R&I value proposition.

Anticipation
RRI requires assessment of the potential implications and societal expectations regarding R&I.During anticipation, researchers or innovators deliberate with relevant stakeholders on the needs, social desirability and the effects of R&I activities, including the possible unintended consequences.Anticipatory processes respond to "what if" questions that open research agendas to public scrutiny.Anticipation can lead to fundamental opposition to the idea behind a project, and thus further to the possibility of project termination at the outset (Stahl et al., 2017).A review of the literature provides instances of such terminations, e.g. the SPICE project, which was abandoned because of plans for controversial activities, such as geoengineering (Stilgoe et al., 2013).
However, the anticipation that precedes research may not be well founded because of possible inadequate knowledge of the value and potential risks of R&I products prior to their usethe so-called "Collingridge dilemma" (Collingridge, 1982).It has been argued that products are neither intrinsically bad nor good; it is their usage that determines the consequences 2, thereby shaping their nature.In addition, anticipation requires a set of skills.For example, Tavani (2017) indicated the need for "critical reasoning (CT)" in ICT ethics frameworks to help analyze and evaluate arguments that occur in everyday discourse.Regarding the capacity for anticipation, Wilsdon (2014) lamented the existing discipline hierarchy, which has marginalized the contributions of historians, who are skilled in judging the interests that lie behind differing interpretations of the past.Although there is a need for better communication of the future value of existing research activities, through CT or the input of historians, in the context of RRI it remains imperative to listen rather than to silence.Several techniques, such as deliberations or scientific fiction, can be used to elicit stakeholders' viewpoints (Jensen & Vistisen, 2017) to feed into rationales in support of or opposition to R&I activities.

Reflexivity
Reflexivity is an individual's ability and will to reflect on his or her own actions.It is associated with the capacity of actors (researchers and institutions) to question their assumptions, accepted routines and knowledge limitations (Stilgoe et al., 2013;Burget et al., 2017).In the context of R&I, reflexivity means reflecting on the purpose, process, product and people components.It happens in intrapersonal communication and group interactions, and involves a review of experiences, an analysis of causes and effects and the drawing of conclusions (Mezirow, 1990;Høyrup, 2004).It is a practice in which researchers and innovators become "[…] aware that a particular framing of an issue may not be universally held" (Stilgoe et al., 2013(Stilgoe et al., , p. 1571)).Reflexivity in science is also a quest to perform scientific research according to the highest quality standards and to understand the consequences of the researcher's activities (Glerup & Horst, 2014).Researchers find reflexivity an important tool for explaining their research choices and subsequent activities to disseminate their findings, although they may baulk at pressure from policy-makers and funding agencies to engage in it, arguing that it limits their academic freedom (Rosenlund et al., 2017).
There are similarities to the infamous saying: "It is not guns that kill people; it is people who kill people."

Responsiveness
Responsiveness is the process by which the effects of inclusion, anticipation and reflection become visible.According to Stilgoe et al. (2013), responsiveness refers to reaction to, incorporation of and response to inputs, including comments and other forms of feedback, generated during dialogue with stakeholders.Responsible innovation requires the capacity to change or shape the direction of the research or innovation process in response to stakeholders' input, public values and changing circumstances.During anticipation, inclusion and reflection, new knowledge emerges, new research directions can be discovered and unexpected ethical and moral dilemmas can be revealed.Responsible R&I initiatives need to be cognizant of these emerging issues so that they can be appropriately addressed.Projects should be flexible and sensitive to the social dynamics to respond to stakeholder inputs (Owen et al., 2012;Forsberg & Thorstensen, 2018;Klaassen et al., 2017).Holweg (2005) viewed responsiveness as the purposeful and timely reaction to stakeholder inputs or to changes in contexts.This implies the need for institutional and entrepreneurial capacity to shape the process in a desirable, affordable and legitimate way.In the context of ICT, IT solutions (the product) need to be designed and developed in a flexible manner that allows them to be updated and adapted to emerging needs and expectations.

A proposed conceptual model
Figure 1 illustrates an integrated conceptual model of the components and dimensions of RRI that are discussed in depth above.The figure is intended to show how the RRI dimensions can be viewed and apprehended in relation to the RRI components.The outer layer of the model shows the dimensions of RRI.Increased responsiveness in R&I inspires public engagement, which feeds back to the inclusion phase, thereby making the four RRI dimensions cyclic and recursive, with consideration of the purpose, process, product and people.This highlights the iterative nature of working with RRI, which is predominantly about learning at the end of each iteration.The order of the dimensions is not strict, but will entail doing and re-doing activities, resulting in potentially more desirable, acceptable and sustainable R&I outcomes.Careful and thoughtful implementation of the model with shorter or longer iterations should result in accumulation of knowledge and eventual learning around these criteria.

Methodology
The study followed an action case study approach.An action case study is a hybrid method of interpretation, i.e. understanding the stakeholders' views on a project, and intervention, e.g.designing and employing crowdsourcing tools (Braas & Vidgen, 1999).Learning is at the heart of an action case study, with interpretation and intervention alternating during the learning process.
There is an increasing call for research designs that enable interaction with stakeholders to tackle and solve real-world problems (Sørensen et al., 2010;van de Ven, 2007).Ideally, the research design should lead to results with both practical relevance (Sandberg & Tsoukas, 2011) and contributions to theory (Alvesson & Sandberg, 2011).The approach is intrinsically problemsolving, constituting an attempt to learn generally relevant lessons from specific cases and operate concepts and develop comparisons through repeated case applications.Thus, to use this approach in the context of a study on RRI, the "action case" can be said to be a hybrid of understanding the theory-based RRI frameworks and its potential change to practice, including balancing the roles of being a researcher making interpretations of theory to practice and as practitioners (i.e.researchers) involved in the practice of R&I itself.
The overall case for this paper was the publicly funded research project iResponse, running from September 2015 to September 2018.It was a collaboration between Norwegian and Finnish research institutions, universities and private companies, financed by the Norwegian Research Council under the SAMANSVAR and IKT Plus Programme.SAMANSVAR is an acronym that also alludes to "shared responsibility".This programme promotes RRI by explicitly stating that ICT-based research must include stakeholder dialogue to take account of issues surrounding privacy, transparency, risk and inclusion.The iResponse project studied what constitutes socially responsible crowdsourcing through three discrete case studies in Norway on crowdsourcing for urban environment research and planning.
The authors of the paper were active members of the project, including the crowdsourcing activities.Interviews with iResponse project members and relevant stakeholders outside of the project were used as data sources.Furthermore, notes from participatory observations and project documents were synthesized and analyzed to assess the operationalization of RRI in the project.Data were analyzed by a deductive approach (Burnard et al., 2008), that is, through the already developed framework for RRI and our conceptual model (Figure 1).Deliberations over how to interpret RRI in the project context took place at various stages including during development of the project proposal.A first suggested interpretation of RRI for the project was presented at the initial meeting and a first stakeholder workshop was held in September 2015.In early February 2017, during a cross case learning workshop, discussion took place on how to assess the RRI dimensions in relation to the air pollution case study, one of three case studies of the project.
At that point, the project members could discuss experiences and whether the RRI dimensions had been given satisfactory attention and also areas that needed further improvement to put the concept into practice.In May 2017, at the second year project meeting, discussions were held on how RRI was practiced in all three case studies.With these preliminary analyses as inputs, in November 2017, various perspectives on implementing RRI in the whole project were presented in an open workshop.The empirical description in this paper is built on the results of the RRI discussions throughout the project.

The iResponse project
The main goal of the iResponseproject was to explore socially responsible crowdsourcing methods for urban environmental research.The project comprised three main empirical case studies: air pollution mitigation (Case I), stormwater management (Case II) and urban planning (Case III).The outcome of the case studies was access to new and novel data that could facilitate the decision-making processes in these areas.In addition, the whole project aimed at producing appropriate IT solutions and approaches for using crowdsourcing in environmental research.

Mapping of citizens' engagement with digital platforms
Early in the project, an online survey was sent to citizens in Norway, covering 1,933 respondents in the five largest Norwegian cities (see Strandbakken et al., 2017).The questions in the survey aimed to determine the respondents willingness to contribute data to authorities and researchers via digital platforms.The respondents were also asked whom they trusted most to engage in research activities.Most important, the survey helped to assess privacy concerns over the use of digital media.The key findings were that respondents were generally not too skeptical about the use of digital media, including sensors for urban data collection.However, they seemed more reluctant to share information about personal issues and other situations with which they were closely involved, e.g.within the home, than information that could be publicly available.
The results of the online survey gave valuable inputs for the project approach, including that the respondents perceived crowdsourcing as an acceptable method provided that it has a legitimate purpose and an appropriate mode of engagement.The findings suggested that the project consortium members need to invest their time and efforts in meeting these conditions.The survey result was later combined with results of workshops, meetings and interviews with a view to foreseeing unintended consequences of the crowdsourcing activities.

Case I: Crowdsourcing for urban air quality research
Case I was motivated by the need to find up-to-date and precise data sources for estimating the contribution of wood burning for residential heating to urban air pollution.Crowdsourcing was proposed because such data then would come directly from the consumers, with a higher spatial resolution.Crowdsourcing was also assumed to create public awareness of the problem.Stakeholders were continuously enrolled in the decisions on practical approaches and choice of appropriate tools.
In the first stakeholder workshop in 2015, researchers involved in Case I presented the initial idea of developing a mobile app that enabled weekly reporting of wood consumption and stove type.The first suggestion for the app was that users should submit their location, in addition to their consumption data.Stakeholders at the workshop expressed concerns about the risk that their whereabouts could be traced without their knowledge if identifiable information such as names and addresses were given, thus potentially infringing their privacy.Consequently, they advised consortium members to collect postal codes instead and to develop a web app.This subsequently shaped the R&I product so that it was less invasive (advantageous for society) but offered lower spatial resolution (disadvantageous for researchers), thus implying possible trade-offs in conducting research within an RRI framework.The stakeholders in the workshop also emphasized the importance of an engaging theme for the research of crowdsourcing were to be a viable method of pursuing it.They indicated that reporting wood-burning consumption could be mundane, thus rendering the theme less than ideal for stimulating social engagement in R&I.
Rather than asking for the public's direct involvement, stakeholders suggested alternative methods for data collection such as sensors in chimneys and data collection from agencies and online sources.These suggestions were feasible; however, they would not address the need for broader engagement of the public in R&I activities.There was also doubt as to the usefulness of crowdsourced data, as it is not statistically representative (Fossum et al., 2017).The suggestions inspired the development of three ICT-based crowdsourcing approaches, instead of just the one stipulated in the original project proposal.Accordingly, the mobile app was designed as a web application with a postal code as the entry point, following national data protection guidelines.Recognizing the potential challenge of involving citizens in collecting and reporting data, the researchers used existing sources to start building their dataset for estimating particulate emissions to the air.Web crawling from a Norwegian classified advertisements website containing listings of real estate was also done to determine the kinds of stoves that were most frequently used in the areas under study (Lopez-Aparicio et al., 2018).After discussions in the consortium, the Public Participation Geographic Information System (PPGIS), originally planned to be of use only in Case III (see below), was used experimentally, which provided quick and valuable results relevant for the case.
Citizen participation was considered one of the main challenges in this case.In general, people were acknowledged as being too busy to choose to engage continuously in research and data reporting.However, alternative crowdsourcing pathways emerged from the stakeholder interaction that diversified the crowdsourcing approaches.As such, it can be argued that RRI shaped the crowdsourcing process in this case in a more socially responsible direction.On the other hand, the researchers reported that responding to stakeholders' various inputs and questions required a significant amount of resources.For example, explaining to people outside the community of experts on air pollution why certain types of datasets were better and more desirable than others was found to be difficult and time-consuming.RRI as a concept was also considered as something new and not sufficiently established for smooth and easy practical implementation in the case study.This indicated a need for further reflections in consortium meetings on operationalization of RRI in the project.The knowledge gained from this case was shared and discussed with Case II in a cross case learning workshop, which enabled anticipation and reflection on possible challenges for crowdsourcing, such as sufficient participation rates, data representativeness and questionnaire design.

Case II: crowdsourcing for urban stormwater management
Case II was motivated by a need to tackle the issue of urban stormwater.The management of stormwater, i.e. surface water from heavy rainfall events and snow melting, in urban areas in particular, is an evolving field that has been the subject of increased academic and political interest, especially in the light of increasing urbanization, as well as the recent heavy cloud burst events in cities, such as the one in Copenhagen in 2011.State-of-the-art research has sparked interest in using stormwater as a resource in the urban landscape through nature-based solutions, the so-called blue-green infrastructure (BGI).
Examples include street trees, storage ponds and open green spaces.Urban hydrological functions are connected to urban nature and landscape design and planning, so that the blue (water) and green (natural structures, parks and plazas) infrastructure can be used for multiple purposes, including flood protection, while providing environmental and social benefits to an area.
Case II originally proposed a hypothesis that citizens were not particularly aware of the damage associated with urban stormwater, including the associated high cost or the potential benefits of the BGI.Hence, unlike Case I, which had a well-defined area of focus at the outset of the project, this case was more open and exploratory in the beginning, so that the dimensions of stormwater management that were applicable for crowdsourcing could be identified.It took a relatively longer time to determine what to crowdsource and to choose an appropriate IT solution.It was more appealing to bring all voices to the discussions in Case II, rather than experimenting with classic crowdsourcing methods.
Meetings and workshops with stakeholders confirmed a lack of public awareness of stormwater problems in cities.Hence, it became apparent that the public was being asked to contribute to research areas in which they did not have any formal knowledge.This led to a realization that the first step in crowdsourcing needed to be public communication on the challenge of stormwater in cities.
Drawing lessons from Case I, the PPGIS tool Maptionnaire was used to collect information on stormwater hotspots, suggestions for additional green spaces and residents' BGI preferences.An additional reason for using Maptionnaire was the stakeholder inputs, which pointed to exploring the potential of existing tools and datasets before developing new methods.Other similar tools and approaches were discovered through discussions with individual stakeholders at their workplaces, thereby increasing knowledge on the topic.The discussions with stakeholders and project members also contributed to the design of a pilot tool that would promote increased awareness of the challenges of stormwater and the role of BGI, in addition to enabling citizens to indicate their preferences regarding BGI design solutions.
Over the course of Case II, important ideas on BGI as a potential solution for stormwater management emerged, as well as questions about the appropriateness of BGI for solving problems in Norwegian and other cities that are already considered to be "green".For example, participants brought up the consideration that tree foliage might contribute to the potential clogging of sand traps and drainage systems and as such to a potential increase in stormwater damage unless remedial measures were undertaken.Another point raised was the competing use of urban spaces.The feedback enabled a broader scope when anticipating potential societal challenges and also highlighted the requirement to seek stronger evidence before implementation.In general, the case recognized the interdisciplinary nature of the topic and the need for it to be addressed from the various stakeholder perspectives that emerged in the meetings and discussions (Seifert-Dähnn et al., 2018).
In addition, through recursive dialogue with stakeholders, alternative data sources were obtained and assessed.While some of the data were relevant and accessible, other data, such as data from insurance companies, were regarded to be too biased, as they did not cover all groups of the population.
The learning from Case II was shared with the project members in a cross case workshop, which was conducted in November 2017, with an emphasis on the benefits and disadvantages of using social media for increasing participation.
In sum, deciding on the focal topic and method of crowdsourcing took relatively longer in Case II because of several factors.An integrated approach to urban stormwater management was a relatively recent topic.Engaging citizens in crowdsourcing was even more of a novelty for the municipalities that were the main stakeholders.As the case study progressed, enthusiasm for using crowdsourcing for stormwater management grew, possibly also as a result of growing interest in urban climate adaptation and urban water challenges among stakeholders as well as the public (Seifert-Dähnn et al., 2018).This could potentially also be in part attributed to the ongoing efforts of those conducting Case II.Several individuals, "selfidentified stakeholders", contacted the researchers on the basis of the information available online and contributed to the discussions during follow-up conversations and interactions.
A challenge experienced in Case II was related to internal organizational practice, for example social media had typically not been used as a means of reaching the public to contribute to research.In addition, the kinds of privacy and security issues of which researchers need to be aware when using ICTenabled methods and tools require sound knowledge about information security, which is not an area of expertise for all researchers.

Case III: public participation in urban planning
Case III from the outset proposed the use of the existing PPGIS tool Maptionnaire (cf.Kahila et al., 2017) for gathering residents' opinions and insights related to a specific instance of urban planning.Several areas were explored topically, contentiously and spatially.A Maptionnaire questionnaire was designed with inputs from the stakeholders.This case was even more explicitly dependent on the collaboration and active involvement of city administrators, as they were intended to be the formal issuer of the PPGIS survey to the citizens.The primary purpose of data collection was to contribute to the city planners' decision-making efforts to make the cities greener and more livable, rather than for strictly research purposes only, as for the other cases.
Despite constant efforts, the researchers involved in Case III encountered reluctance to the crowdsourcing activity on the part of the city administrators, which they attributed to poor organizational readiness, including the lack of a single unit that was responsible for the whole policy area.The city administrators were not equipped to manage large volumes of crowdsourced data, nor did they have dedicated organizational resources to meet the residents' expectations in the form of requests for changes to be made in their local areas.Not responding to requests from residents is likely to result in distrust in city administrators.This empirical case study was therefore discontinued.
Case III, however, provided interesting knowledge regarding participatory urban processes and crowdsourcing and thus played a vital role in the anticipation of potential consequences of crowdsourcing.This knowledge was applied to the other cases.In addition, Case III brought to the project a relevant existing IT solution, which influenced the decisions on whether or not to create new tools within the overall project.

Discussion: learning through RRI
The empirical material from this study suggests that practicing RRI can be viewed as a way of conscious learning about the consequences of a particular research project and its activities so that the project may be shaped towards socially desirable outcomes.The use of RRI as a learning tool has not been explored sufficiently in earlier studies.The learning in an ICT project may relate to issues such as risks, privacy concerns and transparency of the research outcomes, processes, purposes and people.These issues are mainly identified through recursive anticipation and reflexivity, which gives inputs to researchers and innovators, enabling them to shape the components in such a way as to respond to the concerns raised by stakeholders.
The iResponse project was set up to meet the data needs of environmental researchers using crowdsourcing as a method.RRI facilitated a learning process on the viability of crowdsourcing for obtaining data for environmental research with sensitivity to potential social consequences.Several stakeholders (people) were involved in setting the research agenda (purpose) (e.g.Case II), defining the research process (e.g.Case I) and introducing innovative crowdsourcing tools (product).
As has been pointed out by Stahl (2013), RRI can be considered as a metalevel responsibility, which coordinates the existing multiple responsibilities associated with R&I activities.In the iResponse project, the overarching responsibility was to contribute to tackling the challenges connected to urban sustainability.Specifically, all three case studies in the project respond to the United Nations Sustainable Development Goals (SDGs) 11: Sustainable cities and communities.Within the RRI framework, nonetheless, researchers are responsible for aligning idiosyncratic perspectives on the value of specific R&I projects to society.Thus, several stakeholders were included so that their perspectives could be mapped, listened to and learnt from, resulting in more desirable and acceptable project outcomes.
While attempting to elucidate the potential effects of crowdsourcing through stakeholder engagement, the project consortium members learnt of alternative crowdsourcing strategies and tools serving the same purpose, but potentially being less socially invasive.Designing the project with an interdisciplinary team and having three relatively confined sub-cases for empirical learning and methodological studies facilitated the internal learning process.Consistent with Magoni et al. (2018), there was a need to diversify the stakeholder engagement methods.For example, as a result of targeted meetings with relevant organizations, the researchers observed a shift in the understanding of the purpose of conducting the research in Case II, which would be difficult to achieve using only stakeholder workshops.It is, however, important to notice that such smaller, subject-specific ("one-toone") meetings carry a risk of siloing and learning that is limited to the knowledge already present in the room, whereas multi-stakeholder meetings often include probing and discussing one another's viewpoints.
While RRI emphasizes inclusive research for learning, it proved to be a challenge to reach and include the public in research activities with a subjectspecific scope.The responsibility for democratic governance of R&I can be regarded as residing mainly with researchers and innovators who have to devise attractive mechanisms of engagement.In this sense, efforts to stimulate and nurture public engagement can be quite demanding of researchers' time and may conflict with the limited time span of publicly funded research projects.Thus, in addition to allocating sufficient time for inclusion, RRI perhaps needs to be complemented by top-down initiatives that create awareness and enthusiasm among the public with respect to participation in research activities.One possible way is strengthening the relationship between public and research organizations, with public organizations, which are generally trusted more than research organizations (Christensen & Laegreid, 2005), taking the lead in reaching out the public.
Another approach would be the use of the traditional survey method.In the iResponse project, we found the survey to be a useful first step in soliciting the opinions of the general public.It is particularly useful for discussing the purposes of R&I.
Anticipation highlights the need for inter-organizational cooperation, to facilitate the sharing of environmental data and new ICT tools.The responsible conduct of R&I is also meant to encompass economic aspects; sharing and identifying already existing research and innovations outcomes could promote the more responsible use of resources, in terms of efficiency and impact.By integrating multiple research processes, it would limit the unnecessary collection of already available data (duplication) and the overburdening of the public.New ICT tools, for example for collecting environmental data, should be designed for flexibility so that they can accommodate future needs.
In the iResponse project, reflexivity motivated scrutiny of the concept of crowdsourcing itself.Questions arose on how to define and understand crowdsourcing (what it is and what it is not), and if there were known ethical, moral and social dilemmas related to it.A thorough dedicated literature review of these aspects as part of the research process was found to enhance further reflexivity.
In relation to responsiveness, the case studies experienced two different events: Incorporation and discontinuations a result of learning about the acceptability and desirability of the research processes and outcomes.While Case I and II shaped their R&I activities to more desirable outcomes by incorporating inputs from stakeholders and the general public, Case III was discontinued, as it was perceived to be undesirable at that time.
In summary, during the operationalization of the RRI framework in a specific project setting, the need for effective communication about the research purpose, processes and products with the public became a central issue.As argued by Wilsdon (2014) and Tavani (2017), there is a requirement for relevant skills to facilitate the operationalization of RRI in practice, including skills of critical reasoning.When it comes to researchers new to ICT-based tools, it is crucial to increase consciousness about IT security in relation to existing tools and systems, including building privacy and security into the design.

Concluding remarks
This paper sheds light on the dynamics of putting RRI into practice in R&I, drawing on empirical materials from a crowdsourcing project for environmental research.The RRI framework played a key role in leading some of the project decisions and actions.Using the RRI framework as an analytical lens, socially responsible crowdsourcing can be described as a learning activity that involves iterative and recursive conduct of inclusion, anticipation, reflexivity and responsiveness on the purpose, process, product and people components.
The paper outlines four key understandings that could potentially influence the practice of RRI in ICT-related R&I in general, and in crowdsourcing in particular.First, sufficient time should be allocated for recruiting and engaging representative participants.This proved challenging in a project with a limited timescale.Second, researchers must have interdisciplinary skills and capacities for communicating the research activities and outputs to stakeholders and the public, as well between different disciplines.This includes critical reasoning and, in the case of ICT-mediated R&I, knowledge related to information security and privacy.Third, design flexibility of ICT tools should be given enough attention, as needs change through time, so that products and related processes are responsive to such changes.Finally, strategically building a good alliance and close co-operation between researchers and public officials is highly relevant for RRI and crowdsourcing projects.Balancing what researchers can offer with what public officials demand requires productive ways of communication, including greater transparency on the part of researchers, particularly in the design phase.
Focusing on public awareness and responsibility could change the trajectory of RRI to one of shared responsibility, in which not only the researchers but also all relevant stakeholders feel an obligation towards the processes and consequences of R&I ventures.Although this paper centered around public participation and participatory approaches through ICT, the findings are applicable to other research domains.

Figure 1 :
Figure 1: A conceptual model connecting components and dimensions of RRI