The implementation of a digital service approach to fostering team autonomy,distant collaboration,and knowledge scaling in large enterprises

Abstract

BACKGROUND:

Autonomous acting of individuals and as teams are key elements of agile, distributed, and partly or entirely distant working environments. The availability of relevant processes, methods, tools, and guidelines is key to leveraging team autonomy.

OBJECTIVE:

This article presents the design and implementation of a digital self-service kit (SSK) approach featuring high scalability, as well as a quality assurance and continuous improvement mechanism. As consumers, the teams within an organization can use these SSK’s anytime and on-demand without any constraints in location, time, or quota. As producers (of knowledge and experience), they can also assume active roles in the extension and continuous improvement of the SSK’s.

METHODS:

This has been achieved in open community networks where feedback is actively leveraged and constantly integrated in the SSK’s design. Such open Communities of Practices (CoP) ensure that all interested parties can contribute to the adequateness of both the content and the provision of the SSK’s in both local and distant corporate settings. Both the design and implementation have been done and evaluated in a large-scale international corporate environment where high cultural diversity, as well as distant collaboration are of key importance.

RESULTS:

The results presented in this article include a generic digital self-service approach to distance learning and coaching of teams in the particular context of the agile transformation of large corporate organizations. Key elements include a strong and systematic expert team involvement in the process of the setup and design of such digital SSK’s, as well as a well-explained and understood kit structure for efficient and effective utilization and re-contextualization of the contained knowledge into team-specific project contexts. This contributes to team autonomy as a major prerequisite for the agile transformation, as well as knowledge scaling across the organisation.

CONCLUSIONS:

The key insights gained from this experiment confirm the high relevance and effectivity of the approach especially during periods where distant collaborations are essential (e.g. during a pandemic crisis).

Keywords

Distance learning knowledge scaling team autonomy digitalization self-service kit agile organisations

Alexander Poth studied computer engineering at Technical University Berlin. He is IT Quality Manager in the Group IT at Volkswagen AG. He is product owner of Testing as a Service (TaaS) and cares about many other interesting things like the Quality innovation NETwork (QiNET) of Volkswagen to ensure that IT service/product development and delivery can fit the quality expectations.

Mario Kottke studied business economics and computer science at Martin-Luther-University Halle. He is an Agile Coach at Volkswagen AG. His working field is the first contact for projects who want to work agile and consulting during the start. His experience is 4 years working as a scrum master and 3 years as an agile coach.

Andreas Riel received an MSc (Dipl.-Ing.) in Telematics from Graz University of Technology, Austria, a PhD in Mechatronics from Vienna University of Technology, Austria, as well as a habilitation (HDR) in Industrial Engineering from Grenoble Alps University, France. He is an independent coach, consultant and trainer in Systems Engineering, Quality Engineering, Innovation Management, as well as Digitalization and Industry 4.0 to major European industry, especially automotive. He is also a part-time Associate Professor at the Institute of Engineering of Grenoble Alps University.

1 Introduction

Over the last ten years, the digitalization megatrend has been driving distant, distributed, and agile working particularly in global large-scale corporate settings [1]. This has led to new approaches to collaborating in teams [2] and the required Information Technology (IT) system support [3]. IT-systems are a key factor in the digitalization and online working [4]. However, any IT-system must be accepted by people [5] having to realize the performance expected from them. To achieve their goals, teams and supporting systems must change many established approaches and procedures [6]. To adequately consider modern organizational challenges, such as: How to foster agile and digital work practices? How to share and multiply knowledge, despite geographical distances and teleworking? How can digital communication and learning leverage organizational development? How to measure and continuously improve team collaboration, quality, autonomy, and performance?

These and other challenges affect several organizational units, in any functions and hierarchy level. This work will focus on the systematic establishment of the Self-Service Kit (SSK) as an approach to support the agile and digital transformation by leveraging on the latter’s mind-set and means. Thereby it also contributes to achieving the companies’ strategic goals. In our definition, an SSK is a combination of different learning and training approaches like web-based training (WBT) [7] and a digital tutorial [8] provided by domain-experts to many –in general –geographically distributed users [9]. The selection of the approaches has been made to enable the teams to solve the specific issue addressed by the SSK in a digital and pull-based manner without team-external support (like coaches or trainers) and by providing the relevant artefacts. A WBT facilitates the delivery of specific knowledge to people needing or asking for it. A self-service provides a workflow to ensure that some specific outcomes are produced [10]. The integration of both, a WBT and a self-service approach, leads to an SSK which is delivered in a workflow pulled by the consumers supporting them by offering relevant knowledge and artefacts to produce specific outcomes and ensuring guidance during the outcome production workflow. The difference to a pure self-service is that the latter is typically a pre-defined (online) workflow without options for consumer specific individualization. In our understanding, a SSK shall not only enable people to work with the suggested artefacts by offering application knowledge and background information, but also enable them to tailor things to fit into their specific team setting and working environment. This adaptability is considered key to success when it comes to fostering knowledge build-up, distribution and change within diverse organizational cultures. Given the rapid evolution of knowledge in our connected digital era, any organization must ensure that SSK’s have up-to-date content delivered in the way that is most appropriate within a organizational and environmental context. For this, SSK’s need to be embedded in a life cycle management [11] for developing, delivering, and maintaining SSK’s.

In our view, the COVID-19 pandemic pushed the relevance of the SSK approach to a next level due to the significantly increased need for teleworking. A distant support readily available at any time and on-demand to distributed employees having to collaborate and learn, can provide a significant lever to helping these people adapt to the new working conditions and environment faster and in a more sustainable fashion [12]. Furthermore, the SSK approach also helps identify actions for more resilient working environments for collaborative online working and empowerment. The information retrieved from the pandemic impact on the SSK delivery can be a valuable quality indicator about the established SSK delivery approach.

When developing and implementing the SSK concept, this work takes focus on the quality management perspective on knowledge distribution, scaling, and agile transition. In this article, the term knowledge scaling in a large organization focusses on sharing knowledge within organizational units as well as across them and beyond. In the IT context, scaling is a term used for an approach to delivering (providing) something fast to a large number of users –in case of scaling knowledge, the distribution of the knowledge has to be fast for example by parallelizing the multiplication of knowledge. Scaling is distinguished by small-, large- and very large-scale and is defined in an agile context by the amount of involved teams [13]. An agile transition is the way of an organization to adopt agile approaches like Scrum and their practices and rituals. Typically, it is a long way and often started without a predefined termination date. A result of an agile implementation is that autonomous teams are established. An autonomous team is defined by self-organization which includes the ability of learning to learn. However, autonomy does not come for free but rather with costs linked to difficulties aligning team and avoiding redundancies and discrepancies [14]. This perspective implies questions linked with the reliability and ubiquity of access to know-how, processes, methods, and tools, as well as quality services for digitalization and online working, including their continuous update and improvement over time. The case study discussed in this article elaborates on the experience collected from establishing SSK’s in the quality department of the Volkswagen Group IT way over the past three years, including the period of COVID-19 pandemic.

2 Related work

While we could not find a concept comparable to SSK’s in literature, a lot of related contributions have been published that can constitute building blocks to create a SSK framework. This section elaborates the context for the SSK approach by introducing such building blocks and the related concepts. Human Resource Development (HRD) includes any activities or processes that have the potential to develop work-based knowledge or expertise for personnel, group, team, community or organizations gains or benefits [15]. Based on their definitions, the SSK approach contributes to HRD. Knowledge sharing in large organizations is a non-trivial task [16] and needs support by tools like enterprise social network sites [17]. The differences between agile and tayloristic organizations in knowledge-sharing is that the agile organization uses more intensive socialization by collaboration etc. instead of explicit knowledge externalized in documents etc. [18]. Large scaling success is based on frameworks [19] which have to be supported by the executives and human resources [20] for sustainable success in both cases: bottom up and top down implementation. As autonomous teams are a key element in agile organizations for product development [21], the development of the SSK has to support team autonomy and develop the people by using SSKs towards autonomous working. Furthermore, agile teams are based on the specific competencies of their individuals [22] which have to be developed by HRD. This includes that in agile environments people are key success factors for the outcomes [23]. Especially in organization with a high degree of digitalization distributed working becomes normal and have to cover the team dynamics of distributed agile teams [24]. The building block set of SSKs is inspired by the following leaning and training approaches.

Learning-by-doing is a useful approach in practice and industry [25]. SSK’s foster the learning-by-doing method based on goal-based scenarios [26] by adopting a guided approach through the combination with other learning concepts, particularly blended learning. There exist many different blended learning approaches [27]. In this context, the focus is mostly on self-paced and asynchronous formats [28, 29] extended by synchronous online formats for the online meetings of groups to work together on a topic [30].

Furthermore, Problem-Based Learning (PBL) [31] is a topic related to SSK’s because the latter address particular problems while supporting the SSK applicants in solving them. Approaches to providing guidance are analysed in [32]. The SSK, however, does not pose a particular problem but rather provides the appropriate set of questions to ask to identify a problem in practice, and leverages on this problem identification process to propose methods that help in the problem resolution process.

Another relevant aspect of the blended learning is the e-learning aspect of labs [33]. Labs are used for practical training guided by instructions. However, labs are experimentation environments that normally represent only a limited set of real-world scenarios and their contexts. In the case of SSK’s, the lab is replaced by the real-life context. Therefore, it is important that both the problem identification and the solution guidance is appropriate to avoid significant failures [34] leading to harm [35] either by misguidance, misuse or even by accident. Consequently, and according to Bloom’s model of learning [36], the minimum SSK objective is “applying” rather than “understanding” or even lower, which is typically the minimum learning target for Web-based trainings (WBT). WBT are established approaches to train people online. While WBT’s transfer knowledge [37], they do not have the objective of guiding the transfer and re-contextualization of the transferred knowledge to a specific task or entire project. From that perspective, SSK’s have learning objectives and maturity expectations that are significantly superior to common WBT. These further augments the need of setting SSK’s into an adequate design [38] context, which depends on a lot of influencing factors.

Tutorials are a common learning approach particularly in engineering [39]. They can be built systematically [40] and supported by software tools [41]. There exist numerous approaches to establishing tutorial-based education using digital media, e.g. [42]. These approaches are systematically analysed to develop online tutorials [43]. However, it is not trivial to establish a tutorial in a problem-based learning environment [44], and special care is needed during implementation and deployment [45].

Given the self-service constraint of our approach, a kind of Personal Learning Environment (PLE) [46] is needed for learners to be motivated to achieving the required learning level [47]. From an SKK design perspective, this is not always evident, since SSK’s are supposed to address a large audience in a large-scale enterprise setting. Furthermore, industry projects are increasingly team-based, which leads to the requirement for personalizing for team culture rather than individualizing. Including wikis [48] in SSK’s are one quite straightforward way of implementing such a collaborative learning environment. However currently no established systematic pedagogical quality analysis of SSK is available like for specific online tutorials [49].

Team autonomy in large-scale corporate organizations is efficient if goals are well defined and transparent on a team level [50]. For SSK’s to be most effective, this implies that they support the specification and achievement of goals. Furthermore, autonomy and self-organizing teams come together and need cross-functionality, which is based on sharing of knowledge [51] that is available both within and outside the teams.

3 Research objectives

Based on the literature analysis and practical experiences from large-scale industrial domains, this section structures the objectives to be achieved in requirements and constraints which any SSK approach shall cover. Based on them, organizations can design and implement different SSK methodologies, dependent on their specific needs and objectives. Based on the demand context we have introduced and related published insights, we define the following fundamental generic objectives and requirements to the SSK design:

Empowerment of teams through SSK’s: to accomplish their specific work in adequate quality, teams need skills and capabilities, regardless of their organizational and/or geographical settings [52].

Accompaniment, guidance, and facilitation of teams on their agile and digital transitions [53].

Leveraging team autonomy as a key asset of the agile mind-set, where mastery implies the capability of accomplishing work autono-mously while assuming responsibility for the outcomes [54]. The importance of this asset is even emphasized in distant, virtual teams.

Leveraging unified team practices across distributed teams. Any organization must improve and learn from their projects to reach their best possible performance [55].

Leveraging distributed team learning [56] while minimizing the need for local presence of trainers and coaches as limited resources, particularly in teleworking environments.

Leveraging mutual knowledge exchange thanks to structured guidance [57]. This process shall happen through the collaborative involvement of teams rather than through individual studying and learning.

Leveraging agile culture maturity improvement of an organization [58]. Agile autonomy and mastery come together when teams ensure and demonstrate maturity for the tasks they have to perform.

Ensuring sustainability and continuous improvement of the above levers compliant with the established Deming Cycle [59]. This continuous improvement shall be applied both to the approach itself, as well as to the content it provides.

SSK quality assurance is essential due to the (desired) large-scale effect SKKs shall provoke [60].

In the context of agile quality management in large-scale corporate settings, we impose the following constraints to the design of SSK’s:

During the agile transition, the teams are often not end-to-end responsible [61].

Regulation and compliance break the end-to-end responsibility [62].

Teams have their individual transition speeds, each requiring different support actions [63].

Teams develop products and services of diverse complexity, which again leads to different support demands [64].

The support demand is also dependent on the teams’ different capability and maturity levels [65].

The support scales with the SSK producer teams’ budgets. Often, initiatives start without dedicated budget, and teams need to find sponsors for each single activity, focusing on the actual value proposition [66].

Even without dedicated budget and headcount, the support must scale [67].

4 Methodology

To develop the SSK approach in the context of a large-scale corporate setting, we have chosen the design-science methodology [68]. Starting with the requirements and design constraints derived from literature and practice, we developed the SSK concept and structure in an iterative way. As design-science research is similar to action research [69] we combined them to fit better to the practical environment of the SSK development. The diagnosing leads to the problem awareness which is given in the issue that not everything can be done by coaches, mentors or trainers and especially agile coaches are rare. The objective is to reduce the effort for “repetitive” coaching parts and focus on “special cases”. An approach has to be designed which fits to agile teams and their organization. This leads to the derived requirements for the SSK as an approach to ensure scaling agile by the constraint of limited coaching and supporting autonomy of the teams. As part of the action planning we designed the artefacts filling this structure with concrete learning elements one-by-one in close collaboration with domain experts and the insights from the analyzed literature for our suggestions for the artefacts. Where possible existing artefacts are used as building blocks for the SSK to reduce development effort and build on a proven in use fundament or/and reliable research results. In this process, we placed particular emphasis on each SSK element’s practical relevance in the concerned team settings and life cycle stages, as well as their neat integration with other existing artefacts and modules during the artefact development. The action taking was made by application/evaluation and the evaluation of the result artefacts [70] were subsequently applied in real-world teams and projects whose feedback we exploited to improve and adjust the artefacts as part of our learning to conclude the results of the evaluations and actions, as well as the concept itself where relevant. In this article, we will not include any technical feedback evaluation. Instead, we will only focus on observations, experiences and surveys related to the teams’ learning efficiency and satisfaction level. This publication is also part of the communication of the outcomes [71].

4.1 SSK package

We define the SSK package as a set of artefacts that enable their users to acquire practically applicable knowledge about a specific topic addressed by a particular SSK. Based on Bloom’s model, the SSK package is designed to support at least applying of the transferred knowledge to the teams’ project settings with their specific product- or service-contexts.

To empower the teams adequately, we distinguish two SSK package types. The first type is the technical SSK package, which helps teams tackling specific technology challenge in their products or services. Typical package artefacts are questionnaires and checklist templates covering specific technology aspects in the application domain, as well as tools. Technology topics for technical SSK’s for are e.g. Testing as a Service (TaaS) [72] and Serverless Computing [73]. The second type of SSK’s are methodical SSK’s facilitating the adoption of team behaviours and/or the generation of relevant outcomes using specific processes and/or methods that are imposed by the corporate governance structure. Typical package artefacts are workflows for applying methods and “how-to’s” explaining the adoption of each methodical step. A topic for methodical SSK’s are e.g. Levels of Done (LoD) derivation [74]. As these two types deliver different artefacts, their designs are also different. To ensure that the teams get the desired empowerment level, the SSK design teams (producers) pay close attention to the required delivery artefacts.

To ensure mutual knowledge exchange in real-world corporate environments (R6), SSK packages must deliver the artefacts ready for use in daily work practice. This includes templates or checklists, however also additional background information about the creation of these artefacts and the resulting objectives, prerequisites, and constraints. Especially in complex project environments, it is vital to know why particular technical or methodical content is provided in the SSK. This gives the consumers the autonomy to decide about the adoption or rejection of some parts of an SSK, enabling the teams to reach higher Bloom levels (R3, R7).

To ensure continuous improvement of the SSK’s artefacts (R8), all consumers can give feedback to the SSK producers. Authors of the SSK are listed on the SSK and be contacted directly. Additionally generic email-boxes can be addressed, too. Furthermore, SSK producers request actively improvement by consumer feedback in trainings or coaching sessions, which also give producers the opportunity to collect returns on experience based on team observation. Both these kinds of feedback loops are particularly important in the initial development phase of an SSK (R9).

Any SSK artefacts are designed for a fully digital delivery (R1, R2). Consequently, the digital formats used must be commodities to enable as many consumers as possible to use the SSK. Furthermore, the concepts and methods of the SSK must be integrated into relevant “classical” trainings as well. This is needed to offer consumers lacking digital mind-set the opportunity to also benefit from the developed artefacts. In addition, training content must be aligned with SSK content continuously.

The fundamental SSK artefacts are as follows (Fig. 1):

Fig. 1

Artefacts of an SSK package and their content.

A short introduction including:

Information about the purpose of the SSK;

Overall instructions how to use it;

A summary of the SSK’s contents.

The core working artefacts are materials that shall be used to produce the desired outcomes, or otherwise the outcomes themselves in the form of templates, spreadsheets for checklists, as well as documents and vector-graphics that can be scaled to posters to facilitate interactive team work.

Background information about the SSK providing answers to fundamental or frequently asked questions linked to the SSK’s motivation, its purpose, its producers and supporting communities, etc.

All SSK artefacts have a date/version/history information for quickly identifying updates and content providers.

4.2 SSK life cycle

We define a specific life cycle for SSK’s in order to account for primarily the quality requirement (R9) we have identified for large-scale corporate environments:

SSK life cycle phases are linked to specific quality objectives that need to be achieved and achievement levels continuously monitored.

SSK life cycle phases provide the fundamental basis for a holistic governance of SSK’s.

Any SSK life cycle shall contain at least the following four phases (Fig. 2):

Fig. 2

Basic life cycle of an SSK.

Development –initial setup of a new SSK as delivered by its producers (“version 1.0”). This includes the analysis of the initial situation, the development of the artefacts and their simulation, as well as their validation in proofs of concept and piloting. Only mature artefacts—sometimes after a few iterations [74]—may be bundled to the SSK version 1.0.

Deliver –a stable, proven set of artefacts is delivered as an SSK version. In general, consumers can use any SSK version without any dependency to the SSK producers.

Update –a new, improved version of a SSK issued from cyclic checks concerning the SSK’s compliance with the state-of-the-art, as well as from integration of feedbacks.

Retired –in case a SSK reaches obsolescence (for whatever reasons), it shall be retired (R9). This will typically happen if the state-of-the-art requires a completely new SSK, or if a disruption makes the existing SSK obsolete. A retired SSK does not get any updates and cannot be newly instantiated for use by consumers (unless there is a justified reason for that, e.g. legacy projects).

Producers carry out the development and update phases, while consumers have key roles in the delivery and retired phases. During the update phase, the existing version of the SSK is still usable, however a new version of the SSK will be the outcome.

The update phase is key to continuous improvement of SSKs (R8). Any SSK must be regularly investigated against changes of its application environment. Particularly, technical SSK’s can be embedded in a fast-moving environment. Relevant changes must lead to an SSK update. Changes and related feedbacks including their originators shall be included in the SSK’s version history. This shall contribute consumers to contribute actively to SSK improvement. Depending on the SSK, the update cycle time varies, however, from experience we recommend at least one update cycle per year (R9).

4.3 SSK delivery

The SSK delivery signifies the standardized procedure that makes SSK’s accessible for consumers. It mainly covers any technical aspects for digital delivery. The delivery procedure maps to the deliver and retired life cycle stages. The latter retired stage is affected by delivery in a sense that consumers shall be triggered to delete their local copies of any SSK “delivered” as retired.

To enable consumer-pull based digital SSK delivery, any SSK content must be provided in digital form on internal platforms like social media, wikis or share-points. To leverage team autonomy, the delivery procedure must scale with the consumer demand and be available anytime (24*7). To facilitate quick retrieval of SSK’s and artefacts relating to a particular problem or topic, a search engine and/or a semantic structure linking related SSK’s together shall be provided on top of any SSK. In addition, it is useful to have search tags or full-text search available were possible. Furthermore, in large enterprises special exist frameworks for agile transitions, as well as dedicated topic-related methods and toolboxes or collections. In these cases, the integration of SSK delivery into specific frameworks is also important, because many consumers will search in these topic locations first.

To leverage distributed learning with the SSK’s (R5), the delivery platform shall facilitate looking up relevant SSK’s, and provide dedicated background information that goes beyond the pure SSK “application”. A full-text indexed search can deliver results from other topic domains and thereby help relating or linking knowledge between different expert areas including their human knowledge providers (possibly including consumers as well in order to link stakeholders working on similar projects and/or facing similar challenges).

To ensure continuous improvement of the SSK’s delivery (R8), the delivery platform must support easy and direct feedback for the consumers. This will contribute to motivating consumers to provide feedback on a regular basis.

4.4 Overall SSK approach

The overall SSK approach as depicted in Fig. 3 focuses on a holistic delivery experience by the consumers and ensures added-value and integration also on an organizational level. The SSK approach covers aspects of governance and strategic alignment of the individual SSK packages.

Fig. 3

From an SSK artefact to delivery as a holistic SSK approach.

Accompaniment and guidance of the team development (R2) is supported by providing expert knowledge by the producer-team to the SSK consumers. In particular, the additional background information for adopting the SSK goes beyond the typical “how-to” services. Furthermore, consumers can use the SSK as individuals or in small groups to transfer the knowledge in a consumer team. SSK’s need a strong alignment with organizational training programs to ensure that people get properly trained for the use of SSK’s and are aware of the available SSK topics. To establish the link between the traditional training offers and the SSK ‘s, the SSK producers have to identify the relevant trainings and define anchor points in the training curriculum to the SSK. Moreover, established trainings can use the SSK as a didactic hands-on part of the training because everything is ready to use and helps the training participants to transfer their learnings to practical exercises, resulting in a win-win situation.

To leverage unified team practices (R4), the SSK approach establishes the generic fundamental modules package, life cycle and delivery. This setting ensures that the SSK producers who want to contribute to the SSK approach must accept the SSK design rules and guidelines. This helps to establish a set of quality criteria and enforce them by not publishing or retiring SSK’s which are not aligned (R9). To ensure a high-quality level of each SSK, a team of producers with expert knowledge in the domain of the SSK’s focus is a fundamental prerequisite for creating an SSK. High quality is essential because each SSK is delivered to a wide range of consumers and will establish unified knowledge and practices within the entire organization (R9). Furthermore, a checklist (Table 1) and templates guide and support SSK producers during SSK production.

Table 1

Checklist for the SSK development by the producers

ID	Topic	Done
1	Scope of SSK is defined
2	Producer team for SSK development is established
3	SSK requirements/constraints are identified
4	Development approach for SSK is selected
5	Evaluation approach is defined
6	Results and lessons learned from Evaluation are integrated
7	SSK is based on “SSK template”
8	SSK is “published”

To leverage the agile culture and mind-set (R3), the SSK approach has been designed to support the target state of an agile transition with autonomous teams. From the quality perspective, autonomy is coupled with mastery. Only if people are empowered to master their tasks, they will be able to assume the responsibility of their actions and work with autonomy without the risk of creating big harms by accident. Another important aspect of autonomous agile teams is that they scale easily. The SSK approach supports team autonomy thanks to the pull principle and its timely unlimited digital availability. An additional benefit of the agile culture and its autonomous teams is the resilience of the organization in case of unexpected hazards. The SSK’s pull principle supports the agile transition from traditional hierarchy push to autonomous team pull.

To ensure continuous improvement of the SSK approach (R8), a structured governance of SSKs is also required. For example, governance must identify in which domain/context an SSK shall be provided in the future, what the moving state-of-the-art is, and how feedbacks shall be integrated. Although SSK producers will care for some of these aspects, however not necessarily by self-organization. Especially new topics need this active governance because they do not have established producers who can address them in a self-organized way. In addition, in case of failing self-organization, the governance can trigger SSK updates or enforce their retirement if there are no producers willing to care about a released SSK. The principle behind the governance is to have an instance that acts only if needed to ensure quality (R9) as well as updates to the state-of-the-art. Normally, domain experts, i.e. potential SSK producers, should know that they have the responsibility to share their knowledge with others in their organization and see the SSK approach as an option to scale their expertise in an efficient way. However, since not every corporate culture has achieved that “ideal” state, the SSK governance should have a strong alignment with existing governance departments to have their support by “referencing” to the SSK’s as an amplifier for the awareness and relevance about the SSK topics in the teams.

For identifying topics that may become SSK-relevant in the future, cooperating with new organizational units can be particularly fruitful. Their fast learning and growing curves force them to work in a “naturally agile” way. They are interested to solve their quality issues, knowledge sharing topics etc. Typically, other teams face similar challenges, however their organizational context is much more constrained by legacy and organizational cultures that have grown specific traits over several years. Since new organizational units often have interfaces to established ones, this can lead to a win-win situation in which the new units solve their internal issues with SSKs and make the work on the interfaces easier if the established organizations learn from the new unit’s working style via their SSKs.

Another way of planning future SSK’s is by technology foresight. Cooperation with various legal group entities, as well as external partners such universities and high-tech start-ups can help identifying upcoming technologies in time and use SSK’s to introduce them gradually but thoroughly in the organization. It can also inspire the improvement/replacement of existing technologies, processes, and methodologies.

The structured approach comprising SSK artefacts, the SSK’s life cycle and delivery model supports immediate deployment in a small context as well as scaling up within large organization level:

iterative and incremental extension of SSKs (inherently agile delivery approach);

no significant planning efforts and budget needed (inherently agile delivery approach with a “story” and their resource allocation for the tasks);

direct value delivery by “first consumer”-pull;

systematic integration into relevant communities;

digital and online delivery by design for scaling.

These benefits provide the base to enable mostly every organization to start their SSK initiative to support their digitalization programs and their transitions to autonomous agile working.

5 Evaluation

This section first describes the environment that served as the instantiation and evaluation context for the proposed generic SSK approach. Then, it presents the evaluation method and results.

5.1 SSK concept instantiation context

The instantiation context is the Volkswagen Group IT. The Agile Center of Excellence (ACE) supports the agile transition of the Group IT and other business areas through its coaching and training services. The Quality innovation NETwork (QiNET) as part of Test & Quality Assurance (TQA) has the objective to develop and establish new approaches related to IT quality management topics. Both are working together to facilitate the organization’s transition to a more digital and agile working environment. As both have been facing the same issue that not all facilitation requests of project teams can be served with coaching’s, they had to establish a complementary approach. The entire organization is scaling faster in size and geography than an individual department like the QM department can scale. This implies that coaching with its linear effort per scaling team will fail.

Furthermore, topics like quality management and assurance with their traditional approaches based on frequent checks, audits etc. have an acceptance issue in agile environments characterized by an autonomy mind-set. This leads to the objective to change quality activities from a push to a pull, which fits much better to the working style of an autonomous team. Moreover, the quality activities must scale with the speed of the organization’s team transitions to agile to avoid being a bottleneck with the quality activities. Additionally, is in scope to ensure that in transitions is trained mastery in quality as a base for the team autonomy. To support the teams’ demands by their pulling requests with a high-quality standard a SSK based approach is used because the scaling of quality experts is possible to a limited extent only.

5.2 Specific instantiation aspects

At the beginning of the introduction of SSK’s the change management perspective led us to start with a couple of “new” topics which did not have established supporting offers like trainings and coaching. The objective was to make product and service teams create value themselves without going into competition on market shares with the established offers. This ensures reducing SSK consumer reluctance by comparing the established offer with the SSK offer. Furthermore, the established coaches consider the SSK offer an enhancement to their established support services rather than a competition to their work. Table 2 shows SSK topics which are currently in the life cycle. They address new ways and approaches related to software quality engineering and management that have not had established coaching and training offers over years.

Table 2
Examples of current SSK’s in the life cycle

SSK topic Focus/supports Quality domain

Product Quality Risk Ideation [74] Organization development Process quality

Level of Done derivation [83] Organization development Process quality

Agile Team Work Quality [84] Organization development Team quality

Team Maturity Model Organization development Team quality

Self-Service Kit development Organization development Product quality

Machine Learning safeguarding [85] Technology usage/adaption Product quality

Blockchain building block evaluation Technology usage/adaption Product quality

Chaos Engineering Technology usage/adaption Product quality

Serverless Sustainability [73] Technology usage/adaption Product quality

Fuzzing Technology usage/adaption Product quality

SSK topic	Focus/supports	Quality domain
Product Quality Risk Ideation [74]	Organization development	Process quality
Level of Done derivation [83]	Organization development	Process quality
Agile Team Work Quality [84]	Organization development	Team quality
Team Maturity Model	Organization development	Team quality
Self-Service Kit development	Organization development	Product quality
Machine Learning safeguarding [85]	Technology usage/adaption	Product quality
Blockchain building block evaluation	Technology usage/adaption	Product quality
Chaos Engineering	Technology usage/adaption	Product quality
Serverless Sustainability [73]	Technology usage/adaption	Product quality
Fuzzing	Technology usage/adaption	Product quality

In practice, some SSKs have dependencies like the agile Team Work Quality as a generic approach can be extended with a domain specific Team Maturity Model for the cloud. In this case, although it is possible to use both SSK’s individually, only applying them together will foster holistic view on the concerned teams. There is a stronger dependency in the case where Product Quality Risk Ideation is a part of the Level of Done concept. This shows that SSK’s must be designed as delimited and entire units or building blocks which can be combined to more complex constructs.

To find the “right” consumers, the SSK’s are announced in the specific communities like the Agile Community (AC) [75], the Quality innovation NETwork (QiNET) [76] and knowledge communities like the Q-circle or the technology’s communities like the Blockchain community. Furthermore, these communities have a “prosumer” role, i.e., as SSK consumers they also contribute to SSK production. They have the experts for the SSK development team that produces the SSK. This leverages SSK acceptance within these communities since they identify themselves better with “their” product.

Moreover, the communities help identify demands for missing SSKs. An example is the QiNET and its prosumers. Consumers with a subscription of the QiNET come from global locations like Mexico, Spain, Czech Republic and from Group brands like Audi, MAN, Porsche, Seat, Volkswagen commercial vehicles and legal entities like Volkswagen Financial Services or Volkswagen Group IT Services. These experts identify specific future demands and support developing adequate SSK’s.

The current implementation of the SSK approach is established to support the autonomy of the agile teams. The SSK’s are offered and promoted by their relevant communities and are delivered via centralized platforms like the Group Wiki, which offers the agile toolbox. To ensure sustainable governance, the ACE and QiNET work together to integrate the SSKs in their approaches and trainings, which are often conducted together with the Volkswagen Group Academy. This partnership builds the governance of the SSK approach and ensures that cyclic updates are triggered if the SSK maintainers (producers) do not act pro-actively to maintain the state-of-the-art or to integrate feedbacks. SSK’s without active maintainers are “retired” after explicit requesting for maintenance by the governance.

The strong alignment with the ACE also ensures integration into their trainings and coaching, which are enablers and facilitators in the initial transition phase of teams. The ACE is also the anchor to establish a quality management governance within the agile governance. This is a lean solution to have all relevant governance aspects in one hand to reduce typical matrix organizational activities.

Cooperation of the QiNET with universities and newly founded organizational units are the second source to get inspirations about new topics relevant for SSK’s.

5.3 Demonstration of the instantiation

To demonstrate the acceptance and quality of the SSK instantiation at the Volkswagen Group IT, two adoption indicators metrics are used: the SSK view/download numbers, as well as the feedbacks to SSKs.

The views/downloads are a metric which have to been seen in relation to the “market size” of an SSK and the time it has been available. For example, the specific “edge topic” about Blockchain building blocks (see Table 2) will never have the same indicator value as the more “general purpose topic” about Product Quality Risk Ideation with currently more than 500 views/downloads. In such a case, 10 times higher views/download figures are well justified. Apart from this, the Blockchain SSK has been released at a later point of time. View/download figures of the Blockchain SSK and the agile Team Work Quality SSK confirm this observation, since the generic agile Team Work Quality SSK reached scores within only a few weeks that were comparable to the scores reached by the Blockchain SSK in several months. The view/download shows that people are interest on the content of the SSK, however it does not count the actual SSK applications in products or services.

To get more specific information about the SSK approach, a feedback sheet was designed and introduced in the Agile Community (AC) of the ACE, which is a periodic meeting of about 80–120 employees from various organizational units around agile transition topics. This feedback sheet was designed aligned with [77, 78] to identify the impressions of SSK consumers. Its special focus was on finding improvement ideas. The feedbacks were grouped in three blocks. First, a generic block about SSK’s and how SSK’s are accepted as an approach to elaborate outcomes for the work. This block also tried to find out which inspirations SSK’s gave, as well as if SSK’s facilitate teleworking. In the second and third block for technology usage/adoption and organizational development, the feedbacks were focused on application and autonomy support of the team also during COVID-19.

In addition to the feedback sheet’s deployment in the AC, we handed out the sheet to some colleagues in the context of our daily work, even if there is an unknown overlapping with the participants AC participants (they get two “triggers” to give feedback)

To summarize the feedbacks, SSK’s definitely support teleworking, which respondents consider the new state-of-the-art rather than a “new feature” provoking enthusiasm according to Kano [79]. For the COVID-19 related questions, the feedback was similar. SSK’s support working under COVID-19, however it is a second choice driven by the constraints. People prefer working physically together and having real social interactions. An interesting result is that digitalization with teleworking is increasingly accepted as new working standard while formal and structured social distancing is not (yet) accepted. It looks that teleworking is accepted and expected as an option, however not as an obligation.

SSK’s have been developed and established over roughly three years by the QiNET and ACE as a Volkswagen Group wide offer to support the digitalization and develop the agile mind-set in the organization with its structured on-demand learning-by-doing approach. Current observations and feedbacks confirm the establishment of the SSK approach as a useful way to support the digitalization and agile transition.

6 Limitations

This section distinguishes between generic limitations of the current maturity of the developed SSK approach and the limitation of the presented evaluation.

6.1 SSK approach

A key issue is that it is difficult to get feedbacks about SSK’s after their release. This is an issue by design that autonomous teams have also the autonomy to give or not to give feedbacks. In real-world environments, teams have project time pressure and other constraints which considerably limit their time and availability for providing feedback. This constraint leads to pro-active requests for feedbacks by the SSK owner. However, this kind of feedback is not a representative sampling of the SSK consumers. On the other hand, the feedback bias supports the question because mostly extreme opinions are triggers, the positive as the negative ones [80]. As the feedback collection searches for improvement aspects, negative feedbacks are a good source to derive improvement actions.

An option to get more control is delivering their latest version of the SSK only via a platform and restricted “usage terms” which enforce control motivated by the following arguments. The Working Out Loud (WOL) initiative has chosen this option, with the following principal feedback [81]:

Old copies of the guides were being used, so some people were never able to take advantage of the latest developments.

We had minimal information on how Circles were forming and finishing, which affected our ability to improve the method.

The material was being shared with little context, and we found some Circle members struggling to see how the practice could relate to their goals or role.

As our presented SSK approach suffers from similar issues, this kind of feedback “enforcement” could be an option to improve the situation. However, we believe that it will not solve the issue because it does not address the root-cause but rather reveals symptoms only. The handling of the symptoms will lead to “irregularities” in the usage terms. Based on this analysis, we think that this is not the best way in an enterprise environment because every rule (usage terms etc.) must be controlled and enforced, too. We do not want to establish more rules and their governance as needed to keep the organization as lean as possible. We believe that feedback enforcement by “usage terms” approach leads to “irregularities” by design, which is worse than lacking feedbacks [82].

Another important issue is that it is difficult to have a complete portfolio of SSKs. First, it is difficult to predict the demands in the future to have a developed SSK just in time. Second, it is not possible to deliver every kind of knowledge within a SSK. Some knowledge is not offered by “producers”, however it is in the organization. Other knowledge is difficult to offer in the SSK format –complex things which cannot be broken down into generic SSK content and artefacts. Furthermore, there is also an economic aspect behind each SSK –effort to develop and maintain a SSK have to be considered in the Return on Investment (RoI) over its life cycle.

6.2 Evaluation context

The evaluation is a single use case from one global enterprise within different legal entities. The evaluation is undoubtedly biased by the specific enterprise culture, which is mostly European. Furthermore, a specific setting of the enterprise is its multi brand setting, which also implies some specific cultural behaviour in the brands and a “group” culture and behaviour. An additional limitation is the evaluation and observation time of roughly three years to collect information about the experience with the SSK development, delivery and improvement. However, one limitation is that the large organization is triggered by many changes in parallel, and it is difficult to isolate reactions of the organization correlated to a specific trigger variable form the SSK approach. All observations we have made can have side effects to one or more other triggers, as well as on improvement.

The evaluation involved only a part of the organization, because in the large organization it is never assured that all teams know about their opportunities to use knowledge of SSK’s or participate in SSK development.

7 Contributions

As for the main contributions of this work, we distinguish between theoretical and practical contributions in the following two subsections. Theoretical aspects are generic topics which can be developed, enhanced without an instantiation by an organization because it is for example role or concept based. Practical aspects need an organization for application and depends on “real world behaviour”.

7.1 Theory

Digitalization and online delivery need a concept to reach specific objectives such as the one supporting agile working and mind-sets; an established digital platform for offer SSK’s is the base.

The concept must fit to the organizational constraints and culture; the integrated building blocks with their concepts and artefacts of the SSK have to fit, too.

A SSK is a package based on some training material, a procedure with their description, and if needed some tools like templates and background information to enable teams to tailor the SSK to their specific context.

A life cycle management for SSK’s is required to deliver state-of-the-art services; this implies a kind of governance and a long-term ownership.

Consumers/producers can be the same stakeholders in enterprises; this ensures relevance of the outcomes.

It is difficult to say that the offered portfolio of SSKs is complete; not all demands for the near future have been identified.

Getting feedbacks on a released SSK is difficult, as its usage is out of direct control.

There is a research gap for the SSK approach compared to more established concepts like WBT.

7.2 Practice

Not every digitalization approach needs a line organization and large budgets. Finding the right experts, working on SSK’s and offering them in their communities can work well.

A general platform for SSK’s is only an option because often, specific communities within an enterprise can be knowledge hubs too.

Without a service platform, it is difficult to measure SSK’s for systematic improvement.

Based on the SSK for SSK development, a new SSK can be “produced” in a few hours (assumption: all theoretical concepts are developed and practical experience is available in the SSK team).

The approach of the QiNET to work with affected stakeholders to develop SSK’s offers a way for sustainable quality innovation. After three years of successful deployment, the usage figures and results keep growing.

8 Outlook

The presented approach to establishing autonomous teams by supporting and guiding them with SSK’s scales in large enterprises. Furthermore, the approach of having autonomous teams supported by SSK facilities is robust in difficult times like the Corvid-19 pandemic, since the SSK delivery is digital and therefore not affected of physical, hybrid or virtual presence of its producers and consumers. However, some SSK’s benefit from working in small groups around a whiteboard, flipchart or a wall with post it’s, since being physically together is a human need, and digital facilities are weak in fostering team spirit, as confirmed by the feedbacks. This may change over time if people get better customized to teleworking and establish this as a habit in the future digital enterprise culture.

Future research work should further investigate the SSK approach in general since this view is currently lacking in published literature. A special focus should be set on how to ensure systematic feedback and improvement ideas from SSK consumers. Furthermore, it would be interesting to research the life cycle management of SSK’s in different cultural settings beyond our experiences from Europe. Another interesting aspect is how SSK platforms can be established within organizations and without complex administrative overheads.

9 Author contributions

CONCEPTION: Alexander Poth and Mario Kottke

METHODOLOGY: Alexander Poth and Andreas Riel

DATA COLLECTION: Alexander Poth and Mario Kottke

PREPARATION OF THE MANUSCRIPT: Alexander Poth, Mario Kottke and Andreas Riel

REVISION FOR IMPORTANT INTELLECTUAL CONTENT: Andreas Riel

SUPERVISION: Alexander Poth and Andreas Riel

References

Llopis

, Reyes Gonzalez

, Gasco

. Transforming the Firm for the Digital Era: An Organizational Effort Towards an E-culture. Human Systems Management. 2004;23:213–22.

May

, Carter

. A case study of virtual team working in the European automotive industry. International Journal of Industrial Ergonomics. 2001;27(3):171–86.

Ibrahim

, Kadhim Zuwen

AAA

, Alkaraawi

AAI

. KM Embodied in IT for Capable Product Design. Human Systems Management38(2):125–39.

Cleveland-Innes

, Ally

. Affective learning outcomes in workplace training: A test of synchronous vs. asynchronous online learning environments. Canadian Journal of University Continuing Education. 2004;30(1):15–35.

Davis

. User acceptance of information technology: system characteristics, user perceptions and behavioural impacts. International Journal of Man-Machine Studies. 1993;38(3):475–87.

Zeleny

. Customer-specific Value Chain: Beyond Mass Customization?Human Systems Management. 1996;15(2):93–7.

Olson

, Wisher

. The effectiveness of web-based instruction: An initial inquiry. The International Review of Research in Open and Distributed Learning. 2002;3(2):17.

Kelleher

, Pausch

. Stencils-based tutorials: design and evaluation. Proceedings of the SIGCHI conference on Human factors in computing systems. 2005.

Rajabion

, Nazari

, Bandarchi

, Farashiani

, Haddad

. Knowledge Sharing Mechanisms in Virtual Communities: A Review of the Current Literature and Recommendations for Future Research. Human Systems Management Human Systems Management. 2019;38(4):365–84.

10.

Zhu

, Nakata

, Sivakumar

, Grewal

. Self-service technology effectiveness: the role of design features and individual traits. Journal of the Academy of Marketing Science. 2007;35(4):492–506.

11.

Guntamukkala

, Wen

, Tarn

. An Empirical Study of Selecting Software Development Life Cycle Models. Human Systems Management25(4):265–78.

12.

Rastogi

. Knowledge Management and Intellectual Capital as a Paradigm of Value Creation. Human Systems Management. 2002;21(4):229–40.

13.

Dingsøyr

, Fægri

, Itkonen

. What Is Large in Large-Scale? A Taxonomy of Scale for Agile Software Development. In: Jedlitschka A, Kuvaja P, Kuhrmann M, Männistö T, Münch J, Raatikainen M, editors. Product-Focused Software Process Improvement. PROFES 2014. Lecture Notes in Computer Science, vol 8892 (2014), Springer, Cham, pp. 273-276.

14.

Stray

, Moe

, Hoda

. Autonomous agile teams: challenges and future directions for research. In Proceedings of the 19th International Conference on Agile Software Development: Companion, 2018, pp. 1-5.

15.

McLean

, McLean

. If we can’t define HRD in one country, how can we define it in an international context?Human Resource Development International. 2001;4(3):313–26.

16.

Lee

. Knowledge sharing metrics for large organizations. Knowledge Management: Classic and Contemporary Works, The MIT Press, 2000, pp. 403–419.

17.

Ellison

, Gibbs

, Weber

. The use of enterprise social network sites for knowledge sharing in distributed organizations: The role of organizational affordances. American Behavioural Scientist. 2015;59(1):103–23.

18.

Chau

, Maurer

, Melnik

. Knowledge sharing: Agile methods vs. tayloristic methods. WET ICE 2003. Proceedings. Twelfth IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, IEEE, 2003, pp. 302-307.

19.

Conboy

, Carroll

. Implementing large-scale agile frameworks: challenges and recommendations. IEEE Software. 2019;36(2):44–50.

20.

Doz

. Fostering strategic agility: How individual executives and human resource practices contribute. Human Resource Management Review. 2020;30(1):100693.

21.

Patanakul

, Jiyao

, Lynn

. Autonomous teams and new product development. Journal of Product Innovation Management. 2012;29(5):734–50.

22.

Cockburn

, Highsmith

. Agile software development, the people factor. Computer. 2001;34(11):131–3.

23.

Conboy

, Coyle

, Wang

, Pikkarainen

. People over process: key people challenges in agile development. IEEE Software. 2011;28(4):48–57.

24.

Dorairaj

, Noble

, Malik

. Understanding team dynamics in distributed Agile software development. International conference on agile software development, Proceedings, Springer, Berlin, Heidelberg, 2012, pp. 47-61.

25.

Arrow

. The economic implications of learning by doing. Readings in the Theory of Growth. Palgrave Macmillan, London, 1971, pp. 131-149.

26.

Schank

, Berman

, Macpherson

. Learning by doing. Instructional-design theories and models: A new paradigm of instructional theory. 1999;2(2):161–81.

27.

Hoic-Bozic

, Holenko Dlab

, Mornar

. Recommended system and web 2.0 tools to enhance a blended learning model. IEEE Transactions on Education. 2015;59(1):39–44.

28.

Hoic-Bozic

, Mornar

, Boticki

. A blended learning approach to course design and implementation. IEEE Transactions on Education. 2009;52(1):19–30.

29.

Latchman

, Salzmann

, Gillet

, Bouzekri

. Information technology enhanced learning in distance and conventional education. IEEE Transactions on Education. 1999;42(4):247–54.

30.

Singh

. Building effective blended learning programs. Educational Technology. 2003;43(6):51–54.

31.

Hung

, Jonassen

, Liu

. Problem-based learning. Handbook of Research on Educational Communications and Technology. 2008;3(1):485–506.

32.

Hmelo-Silver

, Barrows

. Goals and strategies of a problem-based learning facilitator. Interdisciplinary Journal of Problem-Based Learning. 2006;1(1):21–39.

33.

Dukhanov

, Karpova

, Bochenina

. Design virtual learning labs for courses in computational science with use of cloud computing technologies. Procedia Computer Science. 2014;29:2472–82.

34.

Raspotnig

, Opdahl

. Comparing risk identification techniques for safety and security requirements. Journal of Systems and Software. 2013;86(4):1124–51.

35.

IEC 61508. Functional safety of electrical/electronic/programmable electronic safety-related systems. International Electrotechnical Commission, 2nd ed., 2008.

36.

Bloom

, Krathwohl

, Masia

. Bloom taxonomy of educational objectives. Allyn and Bacon, Pearson Education, 1984.

37.

Williams

. Instructional Design Factors and the Effectiveness of Web-Based Training/Instruction. In: The Cyril O. Houle Scholars in Adult and Continuing Education Program Global Research Perspectives: Volume II. Compiled by Cervero RM, Courtenay BC, Monaghan CH, 2002, pp. 132-145.

38.

Hoic-Bozic

, Mornar

, Boticki

. A blended learning approach to course design and implementation. IEEE Transactions on Education. 2008;52(1):19–30.

39.

Price

, Richardson

, Jelfs

. Face-to-face versus online tutoring support in distance education. Studies in Higher Education. 2007;32(1):1–20.

40.

Antao

, Brodersen

, Bourne

, Cantwell

. Building intelligent tutorial systems for teaching simulation in engineering education. IEEE Transactions on Education. 1992;35(1):50–56.

41.

Wood

. A new approach to interactive tutorial software for engineering education. IEEE transactions on Education. 1996;39(3):399–408.

42.

Korkmaz

, Harwood

. Web-supported chemistry education: Design of an online tutorial for learning molecular symmetry. Journal of Science Education and Technology. 2004;13(2):243–53.

43.

Blummer

, Kritskaya

. Best practices for creating an online tutorial: A literature review. Journal of Web Librarianship. 2009;3(3):199–216.

44.

Papinczak

, Tunny

, Young

. Conducting the symphony: a qualitative study of facilitation in problem-based learning tutorials. Medical Education. 2009;43(4):377–83.

45.

McLinden

, McCall

, Hinton

, Weston

. Participation in online problem-based learning: Insights from postgraduate teachers studying through open and distance education. Distance Education. 2006;27(3):331–53.

46.

Attwell

. Personal learning environments—The future of eLearning? Lifelong Learning. 2007;(2):1-8.

47.

Ascough

. Designing for online distance education: Putting pedagogy before technology. Teaching Theology & Religion. 2002;5(1):17–29.

48.

Engstrom

, Dusty

. Collaborative learning the wiki way. TechTrends. 2005;49(6):12–15.

49.

Kim

, Ko

. A pedagogical analysis of online coding tutorials. Proceedings of the 2017 ACMSIGCSE Technical Symposiumon Computer Science Education, 2017, pp. 321-326.

50.

Moe

, Dahl

, Stray

, Karlsen

, Schjødt-Osmo

. Team autonomy in large-scale agile. Proceedings of the 52nd Hawaii International Conference on System Sciences, 2019.

51.

Hoda

, Murugesan

. Multi-level agile project management challenges: A self-organizing team perspective. Journal of Systems and Software. 2016;(117):245-257.

52.

Hur

, Im

. The influence of e-learning on individual and collective empowerment in the public sector: An empirical study of Korean government employees. International Review of Research in Open and Distributed Learning. 2013;14(4):191–213.

53.

Lamoreux

. Improving agile team learning by improving team reflections [agile software development]. Agile Development Conference (ADC’05), IEEE, 2005, pp. 139-144.

54.

Hoda

, Noble

, Marshall

. Organizing self-organizing teams. Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering, IEEE, vol 1 (2010), pp. 285-294.

55.

Melo

, Cruzes S

, Kon

, Conradi

. Agile team perceptions of productivity factors. Agile Development Conference (ADC’11), IEEE, 2011, pp. 57-66.

56.

Anderson

, Dron

. Three generations of distance education pedagogy. International Review of Research in Open and Distributed Learning. 2011;12(3):80–97.

57.

Chen

, Hsiang

. A study on the critical success factors for corporations embarking on knowledge community-based e-learning. Information Sciences. 2007;177(2):570–86.

58.

Siakas

, Siakas

. The agile professional culture: A source of agile quality. Software Process: Improvement and Practice. 2007;12(6):597–610.

59.

Moen

, Norman

. The History of the PDCA Cycle. Proceedings of the 7th ANQ Congress, Tokyo, 2009, pp. 11.

60.

Frydenberg

. Quality standards in eLearning: A matrix of analysis. The International Review of Research in Open and Distributed Learning. 2002;3(2).

61.

Poth

, Wolf

. Agile Procedures of an Automotive OEM – Views from Different Business Areas. In: Stolfa J, Stolfa S, O’Connor R, Messnarz R, editors. Systems, Software and Services Process Improvement. EuroSPI 2017. Communications in Computer and Information Science, vol 748. Springer, Cham, pp. 513-522.

62.

Poth

, Jacobsen

, Riel

. A systematic approach to agile development in highly regulated environments. 21st International Conference on Agile Software Development (XP 2020), Proceedings, Copenhagen, 2020, doi.org/10.1007/978-3-030-58858-8_12, 9 pages.

63.

Poth

, Kottke

, Riel

. Scaling Agile – A Large Enterprise View on Delivering and Ensuring Sustainable Transitions. In: Przybyłek A, Morales-Trujillo M, editors. Advances in Agile and User-Centred Software Engineering. LASD 2019, MIDI 2019. Lecture Notes in Business Information Processing, vol 376. Springer, Cham, pp. 1-18.

64.

Poth

. Effectivity and economical aspects for agile quality assurance in large enterprises, Journal of Software: Evolution and Process. 2016;28(11):1000–4.

65.

Poth

, Kottke

, Riel

. Evaluation of Agile Team Work Quality, 21st International Conference on Agile Software Development (XP 2020), Proceedings, Copenhagen, 2020, doi.org/10.1007/978-3-030-58858-8_11, 10 pages.

66.

Reis

. The lean startup. Crown Business, New York, 2011.

67.

Ries

. The startup way: how modern companies use entrepreneurial management to transform culture and drive long-term growth. Currency, 2017.

68.

Hevner

, March

, Park

, Ram

. Design Science in Information Systems Research. MIS Quarterly. 2004;28(1):75–105.

69.

Järvinen

. Action research is similar to design science. Quality and Quantity. 2007;41(1):37–54.

70.

Peffers

, Rothenberger

, Tuunanen

, Vaezi

. Design Science Research Evaluation. In: Peffers K, Rothenberger M, Kuechler B, editors. Design Science Research in Information Systems. Advances in Theory and Practice. DESRIST 2012. Lecture Notes in Computer Science, vol 7286. Springer, Berlin, Heidelberg, pp. 398-410.

71.

Peffers

, Tuunanen

, Rothenberger

, Chatterjee

. Design science research methodology for information systems research. Journal of Management Information Systems. 2007;24(3):45–77.

72.

Kösling

, Poth

. Agile Development Offers the Chance to Establish Automated Quality Procedures. In: Stolfa J, Stolfa S, O’Connor R, Messnarz R, editors. Systems, Software and Services Process Improvement. EuroSPI 2017. Communications in Computer and Information Science, vol 748. Springer, Cham, pp. 495-503.

73.

Poth

, Schubert

, Riel

. Sustainability Efficiency Challenges of Modern IT Architectures – A Quality Model for Serverless Energy Footprint. In: Yilmaz M, Niemann J, Clarke P, Messnarz R, editors. Systems, Software and Services Process Improvement. EuroSPI 2020. Communications in Computer and Information Science, vol 1251. Springer, Cham, pp. 289-301.

74.

Poth

, Riel

. Quality Requirements Elicitation by Ideation of Product Quality Risks with Design Thinking. IEEE International Conference on Requirements Engineering, Zürich, 2020, IEEE (2020): 238-49. DOI 10.1109/RE48521.2020.0003

75.

Poth

. Effectivity and economical aspects for agile quality assurance in large enterprises. Journal of Software: Evolution and Process. 2016;28(11):1000–4.

76.

Poth

, Heimann

. How to Innovate Software Quality Assurance and Testing in Large Enterprises? In: Larrucea X, Santamaria I, O’Connor R, Messnarz R, editors. Systems, Software and Services Process Improvement. EuroSPI 2018. Communications in Computer and Information Science, vol 896. Springer, Cham, pp. 437-442.

77.

Brace

. Questionnaire design: How to plan, structure and write survey material for effective market research. Kogan Page Publishers, 2018.

78.

Krosnick

. Questionnaire design. The Palgrave handbook of survey research. Palgrave Macmillan, Cham, 2018, pp. 439-455.

79.

Kano

, Nobuhiku

, Fumio

, Shinichi

. Attractive quality and must-be quality. Journal of the Japanese Society for Quality Control (in Japanese). 1984;14(2):39–48.

80.

Dellarocas

, Charles

. The sound of silence in online feedback: Estimating trading risks in the presence of reporting bias. Management Science. 2008;54(3):460–76.

81.

Working out loud. Blog available from https://workingoutloud.com/blog/new-version-6-of-the-wol-circle-guides, last accessed on 30 June 2020.

82.

Anderson

. Customer satisfaction and word of mouth. Journal of Service Research. 1998;1(1):5–17.

83.

Poth

, Jacobsen

, Riel

. Systematic Agile Development in Regulated Environments. In: Yilmaz M, Niemann J, Clarke P, Messnarz R, editors. Systems, Software and Services Process Improvement. EuroSPI 2020. Communications in Computer and Information Science, vol 1251. Springer, Cham, pp. 191-202.

84.

Poth

, Kottke

, Riel

. Agile Team Work Quality in the Context of Agile Transformations – A Case Study in Large-Scaling Environments. In: Yilmaz M, Niemann J, Clarke P, Messnarz R, editors. Systems, Software and Services Process Improvement. EuroSPI 2020. Communications in Computer and Information Science, vol 1251. Springer, Cham, pp. 232-243.

85.

Poth

, Mayer

, Schlicht

, Riel

. Quality Assurance for Machine Learning – an approach to function and system safeguarding. IEEE International Conference on Software Quality, Reliability and Security, Proceedings, IEEE, 2020, pp. 7.