This post was originally published on January 7, 2022.

Requirements Traceability – How to Go Live

Requirements traceability is required by many industry standards to ensure product quality and safety. The industry standards are based on decades of progress made in systems and quality engineering research with requirements traceability at the core. Benefits from requirements traceability are achieved if and only if traceability is used as a tool during the product development process. These benefits include greatly reduced or eliminated delays, defects, cost overruns, and rework. Here is an overview of the best practice approach to achieve Live Traceability™.

Live Traceability vs. After-the-fact Traceability

Let’s start with some definitions to make sure we are all on the same page. Requirement traceability is defined as tracking the development progress of product requirements from definition and design through development, testing, verification, and validation. There are two forms of requirement traceability: after-the-fact traceability and Live Traceability.

• After-the-fact traceability occurs after the product has been developed and is typically a highly manual effort to try and re-create artifacts to demonstrate traceability that should have occurred during the development process but did not. This effort is undertaken solely for complying with industry standards and satisfying auditor requests for demonstration of process maturity.
• Live Traceability occurs in real time as the product development process progresses to improve overall productivity (by ensuring engineers across disciplines are always working off the most recent and correct versions) and to reduce the risk of negative product outcomes (delays, defects, rework, cost overruns, recalls, etc.) through early detection of issues. The benefits of early detection of issues are significant. Research by INCOSE found that issues not found until verification and validation are 40 to 110 times more costly than if found during design. For this reason, most companies want Live Traceability but are stuck with legacy tools and spreadsheets that do not support it. Since each engineering discipline is allowed to choose its own tooling, the result is a large number of tools with no relationship rules or mechanisms to create Live Traceability across them.

RELATED POST: Requirements Management Guide: Requirements Traceability

So how do you achieve Live Traceability?

Step 1: Define a Traceability Model

Live Traceability requires a model of the key process elements and their relationship rules to monitor during the development process. The systems engineering V Model is a useful framework to start with for data object and relationship definition. Jama Connect® uniquely provides a point and click, configurable, relationship rule capability to enable Live Traceability. Below you see a sample relationship rule diagram from Jama Connect. Relationship rules vary by industry and company-specific requirements. Best practice templates are provided to comply with industry standards and configured to meet client-specific needs. The definition of a traceability model forms the foundation for model-based systems engineering since it defines model elements and their relationship to each other in a consistent manner across the entire system architecture.

Step 2: Setup Continuous Sync for Siloed Tools/Spreadsheets

Once the relationship rules are defined, the next step is to set up continuous sync with best-of-breed tools and spreadsheets used by the various engineering disciplines. The traceability diagram below shows a typical example of best-of-breed tools and where they sync in the Jama Connect relationship model to deliver Live Traceability.

Most companies prioritize the areas of the traceability model that are most prone to lead to costly issues in the absence of a continuous sync. Most commonly, these areas are:

• Software task management – directly linking the decomposition of requirements into user stories enables Live Traceability through the software development process through testing and defect management. The most common best-of-breed tools used are Jira and Azure Dev Ops.
• Test automation – test cases are managed in Jama Connect to align to requirements and ensure traceability across all engineering disciplines with the test automation results sync’d to the traceability model at the verification step. The most common test automation tools are TestRail and qTest.
• Risk analysis (DFMEA/FMEA) – is most often conducted in multiple Microsoft Excel spreadsheets and the assumption has been that Live Traceability was not possible with Excel. Jama Connect is the first requirements management solution to enable Live Traceability with Excel functions and spreadsheets. Risk teams can now work in their preferred spreadsheets AND for the first time achieve live traceability to stay in sync with changes made by any engineering team. Ansys Medini is also a supported integration.
• Model-based systems engineering (MBSE) – the first step in MBSE is to define a relationship model between all product requirements. Once a relationship model is defined, then specifications can be determined through modeling. Jama Connect uniquely provides model-based requirements to sync logically with a SysML modeling tool like Cameo No Magic. Other requirements management tools do not ensure a model-based approach, which most often leads to inconsistent and conflicting fields across teams and projects and provides no coherent relationship model.

Step 3: Monitor for Exceptions

Live Traceability provides the ability, for the first time, to manage by exception the end-to-end product development process across all engineering disciplines. The traceability model defines expected process behavior that can be compared to actual activity to generate exceptions. These exceptions are the early warning indicators of issues that most often lead to delays, cost overruns, rework, defects, and recalls. Below is a view of our Live Trace Explorer that shows you the LIVE state of development for any level of the development project you choose – from the entire cross-discipline effort down to a specific sub-component. Areas of greatest risk appear in red to show where requirement or verification coverage is lacking. Traceability is now a measurement that can be managed and improved with an overall Traceability Score and coverage and verification percentages..

Benefits of Live Traceability

The main benefits of Live Traceability across best-of-breed tools are as follows:

• Reduce the risk of delays, cost overruns, rework, defects, and recalls with early detection of issues through exception management and save 40 to 110 times the cost of issues identified late in the process.
• Comply with industry standards with no after-the-fact manual effort.
• No disruption to engineering teams that continue working in their chosen best-of-breed tools with no need to change tools, fields, values or processes.
• Increase productivity and satisfaction of engineers with the confidence that they are always working on the latest version, reflective of all changes and comments.

LEARN MORE

This post was originally published December 18, 2020, and has been recently updated.

Digital Thread Defined

Digital Thread Definition – a data-driven architecture that links together information generated from across the product lifecycle and is envisioned to be the primary or authoritative data and communication platform for a company’s products at any instance of time.

This is the best definition of Digital Thread we are aware of and is from an excellent 2018 paper by Singh and Willcox at MIT entitled Engineering with a Digital Thread. The term Digital Thread was first used in the 2006 with the publication of the Global Horizons report from USAF Global Science and Technology Vision task force. (If you have an earlier reference please share in the comments). In this document, Digital Thread is defined as “the use of digital tools and representations for design, evaluation, and life cycle management.”

As with many business terms, Digital Thread has now become over-used by consultants and software vendors. The definition of it — and how it differs from Digital Twin — have been interspersed with more general concepts of integration, simulation, data, and analytics and has lost the original, more precise meaning.

Digital Thread Components

Let’s break down the definition of Digital Thread into its components to better understand the concept and share the most common approaches we see as companies move to make the Digital Thread a reality. Here is the definition breakdown:

1 – a data-driven architecture

This recognizes that the use of a single common platform is impossible across all engineering disciplines (software, hardware, electrical, systems, risk, QA, etc.). Instead, a data-driven approach is required that determines the key information required from multiple tools. It’s important to remember that data-driven does not mean “gather all your data” but rather that you should be using data to answer questions. In other words, do not fall into the trap of tool focus, but rather focus on the questions and collect data to provide the answer.

2 – that links together information generated from across the product lifecycle

From initial requirement definition through to product release, significant information is generated across multiple tools. The challenge is to identify what information is most relevant and how to best link the information to make it actionable. The most common link we see is the definition of value to be delivered (user and system requirements). The most typical information captured across the product lifecycle are process statuses and exceptions (e.g., requirements that have not been approved, require rework, or are not fully addressed, gaps in testing or risk analyses). By linking these process statuses to requirements and tracking them through the product lifecycle it is possible to reduce the risk of negative product outcomes (e.g., delays, defects, cost overruns).

3 – and is envisioned to be the primary or authoritative data and communication platform

Most companies refer to this as a “system of record” or a “single version of the truth.” A Digital Thread is much more than simply integration or a data lake. By tying the definition of what is to be delivered (requirements) to the most critical downstream process meta-data, a Digital Thread create the ability to understand the state of the product development process, what risks are visible and what corrective actions should be considered. Without a Digital Thread, a company is flying blind in terms of the risks it faces in product development.

4 – or a company’s products at any instance of time

For a Digital Thread to be truly useful it must always reflect the current state of the product development process. The value is in seeing the product development process for the first time across fragmented teams and tools, to be able to identify process exceptions and early indicators of potential downstream risks. A static database of days or weeks old data will not be sufficient for a process that is changing rapidly across multiple, siloed teams.

Why the Digital Thread is So Important

The product development process is often fragmented across siloed teams and tools which leads to significant risk of product delays, defects, cost overruns, failed verification and validation, recalls, etc. End-to-end process visibility is required for better cross-team collaboration and the early detection of anomalies to reduce these risks. To solve for this, organizations often attempt to force everyone to use one common software platform, forgoing their choice best-of-breed tools. This solution is neither practical — nor particularly realistic — since engineers are (and should continue to be) allowed to choose discipline-specific tooling which optimize their activities.

What is required is a loosely coupled approach that ties together the necessary metadata across these disparate tools in a way that connects the desired outcome (user and system requirements) to downstream activities – the Digital Thread. The Digital Thread is the best approach to reduce the risk of negative product outcomes while preserving engineering autonomy and productivity.

Click here to learn how Jama Connect’s Live Traceability™ enables a digital thread.

LEARN MORE

Traceable Agile – Speed AND Quality Are Possible for Software Factories in Safety-critical Industries

Automotive, aerospace and defense, and industrial companies have largely adopted Agile within rapidly growing software factories to speed time to market in order to stay competitive. These software factories have largely succeeded in speeding up software development for companies within the industries that have adopted it, but maintaining quality is still a key concern. The inability to coordinate development across engineering disciplines has led to product recalls, quality complaints, and has created significant internal challenges to satisfy functional safety requirements from regulators and confidently deliver high-quality software. These challenges — and resulting outcomes — are often so severe that leadership of the software factories have been let go.

Fundamental Questions We Hear

When we ask software factory leaders, “what keeps them up at night?” We consistently hear the following five questions:

• How do I know which product requirements have been missed?
• How do I know which product requirements are not fully covered by test cases?
• How do I know which product requirements have failed to pass tests?
• How do I identify rogue development activity?
• How do I know if changes have been made at the system and / or hardware level that impact the software team?

These are fundamental questions that should be answerable from leading Agile tooling, but they are not. The reason is that Agile tools focus on tasks (define, assign, status, complete, delete) and have no notion of the current and historical state of the project. Tasks are not tied to any state of the project which often leads to drift from the actual needs and requirements of your customer or end user. As a result, these questions are not answerable with Agile tools like Jira and Azure DevOps. Project management tools like Jira Align answer important questions around staffing, sprint planning, and cost allocation, but do not address the critical questions above focused on the real-time state of the software development effort against the approved requirements.

RELATED: What is a Scaled Agile Framework (SAFe) and How Does it Help with Complex Software Development?

The Answer? Traceable Agile.

How do you best speed software and overall product development and still achieve the quality expectations of customers and company leadership? The answer is Traceable Agile. Traceable Agile speeds the FLOW of software development but also maintains the current and historical STATE of the development effort and auto-detects issues early in the software development process. Traceable Agile recognizes that developer activity is best managed as a FLOW using tasks in a tool such as Jira. What is needed to achieve Traceable Agile is to pair a system with Jira that manages the STATE of the development effort at all times. By keeping STATE and FLOW tools separate but integrated, no change is required to software developer process and tools. This is significant. Software leadership can now answer their critical questions without having to undergo a major process and tool change with resistant developers which would slow down development and/or increase staff attrition.

RELATED: How to Achieve Live Traceability™ with Jira^® for Software Development Teams

So how does Traceable Agile work in practice?

Here is an overview and diagram of Jama Connect^® maintaining the STATE of development activity and Jira providing the FLOW.

1. Task activity continues as normal in Jira and risk is auto-detected in Jama Connect by comparing all user stories and bugs in Jira to the expected development and test activity for each requirement in Jama Connect.
2. All exceptions are identified —the ones that answer the questions that keep software factory leadership up at night — such as requirements with no user stories, user stories with no requirements, requirements with no test cases or test results, etc.
3. After the exceptions are inspected in Jama Connect, management can take action and assign corrective tasks in Jira as just another task in the queue for a developer.

RELATED: Extending Live Traceability™ to Product Lifecycle Management (PLM) with Jama Connect^®

This is a fully automated process that leverages automated synchronization of meta data between Jira and Jama Connect via Jama Connect Interchange™. The only metadata that needs to be synchronized from Jira to make Traceable Agile possible is as follows: ID, Created Date, Creator (User), Modified Date, Modifier (User), Title, Status, Link (URL), Relationships. On inspection in Jama Connect of an issue, one simply clicks on the link to go to Jira if more information is required to diagnose.

Many of our leading clients have already implemented Traceable Agile and are significantly improving their Traceability Score™ which we have demonstrated leads to superior performance on quality metrics in our Traceability Benchmark Report.

Feel free to reach out to me to learn more and I will respond.

RELATED: In this video, we will demonstrate and discuss Traceable Agile™
and how speed and quality make software factories and safety-critical industries possible.

How to Plan for Large Language Model (LLM) Adoption Within Your Engineering Organization

The initial free and unprotected access to ChatGPT (the most well-known Large Language Model) has led some individual engineers to try out the tool by using company owned trade secrets and intellectual property (IP) as prompts. The predictable result has been IP leakage with numerous high-profile examples, including at Samsung. As a result, many companies, including Apple, have banned internal use of the technology outright. Additional risks are just starting to be understood given the lack of consent provided by the actual owners of content that was used to train the LLMs. This leaves the concept of ownership of LLM output, and the ability to protect intellectual property that includes LLM output in question and legal experts are advising caution. Clearly, it will take some time for legal frameworks and precedents to be established for the use of LLMs in product development and for enterprise-class integrations to be developed to LLMs that at properly allow for company level standards and governance of IP. Numerous lawsuits, such as Clarkson v OpenAI, are now underway alleging all of the data to train the LLMs was obtained without consent or renumeration and violates copyright law.

RELATED: Best Practices Guide to Requirements & Requirements Management

Clearly, it will take some time for legal frameworks and precedents to be established for the use of LLMs in product development and for enterprise-class integrations to be developed to LLMs that properly allow for company level standards and governance of IP. Given the risks and unresolved legal questions LLMs pose for product development, how should an engineering organization plan an adoption path to achieve potential benefits from intelligent assistance for engineering tasks?

The guidance we provide our clients is to focus on the following three areas, ranked in order of greatest benefit from intelligent assistance:

1. Improve requirements quality – Poorly specified requirements account for up to 64% of defects and are the costliest ones to correct. The International Council on Systems Engineering (INCOSE) and the Easy Approach to Requirements Syntax (EARS) have established best practices for requirements specification and unfortunately, LLMs are trained on publicly available requirements content that is rife with all the most common errors addressed by INCOSE and EARS. The best intelligent assistant to improve requirement quality is a Natural Language Processing (NLP) approach that analyzes requirements against INCOSE and EARS best practices and recommends improvements – which is exactly what Jama Connect Advisor™ does. Jama Connect Advisor protects all IP and engineers learn how to write better requirements through intelligent guidance.
2. Manage by exception – The engineering function is one of the last in the enterprise to be managed through data. The engineering process is often fragmented across teams and tools which leads to late identification of cross-discipline issues that result in defects, delays, cost overruns, and recalls. Jama Connect^® intelligently solves this problem through Live Traceability™ which automatically syncs data across best-of-breed tools and tracks engineering progress against the chosen development model (e.g., V-model) to identify issues as early as possible and thereby reduce the risk of defects, delays, cost overruns, and recalls.
3. Increase engineer productivity – The biggest drains on engineering productivity are most commonly integration meetings and rework. Jama Connect’s Live Traceability intelligently alerts teams to impactful change from other engineering disciplines. Live Traceability eliminates the need for time-consuming and mind-numbing integration meetings and is proven to reduce rework based on our groundbreaking benchmarking study. Further productivity gains can be achieved by leveraging LLMs for requirement decomposition and we intend to be one of the first to market with an enterprise-class solution that protects IP and enables company standards.

RELATED: Best Practices Guide for Writing Requirements

To get started with intelligent assistance, learn how best to improve requirements quality across your engineering organization with the NLP application of EARS and INCOSE best practices.

How to Achieve Higher Levels of the Capability Maturity Model Integration (CMMI): Part 2

Benefits of Live Traceability™

The main benefits of Live Traceability across best-of-breed tools are as follows:

• Reduce the risk of delays, cost overruns, rework, defects, and recalls with early detection of issues through exception management and save 40 to 110 times the cost of issues identified late in the process.
• Achieve CMMI Level 2 maturity for Requirements Management with no after-the-fact manual effort.
• Eliminate disruption to engineering teams that continue working in their chosen best-of-breed tools with no need to change tools, fields, values or processes.
• Increase productivity and satisfaction of engineers with the confidence that they are always working on the latest version, reflective of all changes and comments.

Another core goal of CMMI Level 2 is to involve stakeholders in the requirement review and approval process (see table below). Let’s examine how companies achieve this goal either through meetings or online reviews.

CMMI Level 2 (Managed) Requirements Management

There are two ways to implement this practice: meetings or online reviews. Most engineering organizations still address stakeholder approvals through large and lengthy meetings that involve all relevant engineering disciplines scrolling through the requirements document for feedback. This is a highly inefficient approach that negatively impacts engineering productivity, morale and fails to capture relevant comments, feedback, revisions, and approvals from stakeholders given the format. More mature engineering organizations have brought the review and approval process online to improve the quality and timeliness of feedback, capture all version and approval histories online, and improve engineer productivity and morale. Let’s examine how companies have brought reviews online with Jama Connect® Review Center.

Review Center allows teams to send product requirements for review, define what’s required, invite relevant stakeholders to participate, collaborate, and iterate on resolving issues and approving agreed-upon requirements. By simplifying the revision and approval process, Review Center streamlines reviews and facilitates collaboration, giving stakeholders easy access to provide feedback where required. Jama Connect enables both informal and formal online review processes to support this CMMI best practice.

RELATED: Extending Live Traceability™ to Product Lifecycle Management (PLM) with Jama Connect^®

Formal Reviews

The formal review process enabled by Review Center is shown below:

Review Center enables teams to define a review, invite participants, gather and incorporate feedback from relevant project stakeholders, iterate, track a review’s overall progress, and monitor progress and capture approval signatures if required. Reviewers can respond to a conversation that’s taking place, as well as mark items as “Approved” or “Rejected” to complete the review. Inside Review Center, reviewers can also add electronic signatures to reviews in order to comply with regulatory standards. Jama Connect captures the date and time of completed reviews for auditing, tying each signature to the document under review.

Informal Reviews

Organizations that still want the quality review aspects of Jama Connect but are not bound by producing formal documents of requirements may take a more iterative approach. A “rolling” review is a review that changes the scope of which requirements are included in each revision. For example, each requirement has a “state” field – Draft, Ready for Review, or Approved. In the project side of Jama Connect, requirement owners will mark requirements they feel are “Ready for Review.” Moderators can also edit requirements directly in the review based on feedback from Approvers. Using a Jama Connect Advanced Filter, a review will be started by pulling in only requirements that are marked “Ready for Review.” Using this methodology, the review is much smaller in scope and can typically be completed faster. On a regular cadence, the moderator will review feedback, make changes to requirements as necessary, or potentially update the requirement status to “Approved” if the required stakeholders have approved the requirement. When publishing a new revision, Jama Connect will pull new requirements into the review and cycle out requirements that are “Approved” (these requirements no longer meet the filter criteria of state = “Ready for Review”). This allows teams to review requirements on a regular cadence — or sprint — and cycle requirements into the review when they are ready for feedback and out of the review when they are “Approved.” Almost any item of content you create in Jama Connect may be sent for a review, including requirements, design, test cases, test plans, and test cycle results.

RELATED: Tracing Your Way to Success: The Crucial Role of Traceability in Modern Product and Systems Development

“Review Center is facilitating communication. It has ensured a shared view of the world and agreement from all stakeholders. There are no surprises anymore. Jama Connect enables us to review documents and make decisions easily with everyone coming to a shared conclusion. If we compare it to reviewing the spreadsheets and Word documents versus doing a review in Jama Connect Review Center, it’s about an 80% reduction in time, for sure.” – Craig Grocott, Head of Systems Engineering

To achieve CMMI Level 2 requires defining a development process and adhering to it. Below is a core goal for CMMI Level 2 – evaluate adherence to requirements management process.

CMMI Level 2 (Managed) Requirements Management

Achieving this goal requires the ability to decompose requirements across engineering disciplines and maintain traceability up and downstream as the project progresses with significant changes and rework. Without an underlying system architecture and common data model, this goal becomes unattainable for most organizations. Attempts to manage through Word and Excel, become unwieldy and unable to meet the requirements for Live Traceability, leading to defects, delays, cost-overruns, and recalls. Below, you can see how easy it is to manage traceability and view up and downstream multiple levels in a trace view of requirements in Jama Connect. Jama Connect’s Traceability Model defines the data model across best-of-breed tools to capture actual behavior for traceability and management by exception.

To achieve CMMI Level 3 requires defining a development process and adhering to it. Below is a core goal for CMMI Level 3 – establishing a verification process and adhering to it.

CMMI Level 3 (Defined) Verification

Companies are achieving this goal through Jama Connect by establishing a Traceability Model that requires test verification for requirements and managing by exception through dashboard reporting to ensure verification happens across all requirements. Below is a sample verification dashboard to achieve this goal with customer-specific info redacted. Here you can see how the Verification Leader manages their function through exception management. Specific widgets on the dashboard track requirements without tests, failed tests, tests without requirements linked to verify, bugs without tests, and risks without upstream or downstream traceability. The Traceability Model established in Jama Connect defines the expected behavior against which all activity can be compared to generate exceptions that can be managed through the dashboard. Without this system architecture and data model, managing by exception becomes extremely manual and productivity killing, if not impossible.

CMMI Level 4 requires organizations to have developed predictive scores and benchmarks that enable management to identify product development risk early and remediate at much lower cost than if not identified until late in the development process or after product release into the market. The table below shows the definition of this core, Level 4 goal.

CMMI Level 4 (Quantitatively Managed) Process Performance

Leading companies are achieving this goal by applying Jama Software’s Traceability Score™ and benchmarking engineering projects internally and externally against peer companies. Jama Software is the first to measure traceability thanks to our clients’ participation in a benchmarking dataset of over 40,000 complex product development projects spanning aerospace, automotive, consumer electronics, industrial, medical device, semiconductor, space systems, and more. All of this is made possible by our core product, Jama Connect®, which enables the largest community of engineers using requirements management SaaS (Software as a Service) in the world.

To formally measure traceability, we have established the Traceability Score. The Traceability Score measures the level of actual process adherence to the expected traceability model and can be used to compare performance across projects, teams, divisions, and companies. This score can also determine impacts to schedule, budget, cycle times, risk, and quality.

RELATED: New Research Findings: The Impact of Live Traceability™ on the Digital Thread

Traceability Score definition

Traceability Score = # of established relationships among model elements as specified by the project’s traceability model.

The following diagram provides an illustration for the buildup of the calculation:

1. At the individual requirement level, we can identify each expected relationship defined in a project’s traceability model (i.e., user needs defined by requirements, further refined by sub requirements, and test cases that should verify the requirement, etc.). We can then identify how many of these relationships have been established to get an individual requirement’s traceability.
2. As we go one level higher and measure traceability within a particular element type (e.g., user needs, requirements, tests, etc.) we can sum up the number of expected and established relationships across the set of items, giving us traceability at the element type level.
3. Finally, we can sum up the number of expected and established relationships across all element types, giving us the project’s total Traceability.

Correlations & Hypothesis Test Results

As a process management tool, the value of a Traceability Score is to quantify actual adherence to the specified approach. To determine best practices from the data, statistical tests were run to understand how differing levels of project adherence to Live Traceability can impact desired outcomes. As we have shown, the Traceability Score measures actual adherence to the defined traceability model. The systems engineering discipline, the V model, quality engineering, and more – all rely on the intuition that this approach will yield better results. Anecdotal evidence abounds to support this intuition, but the dataset has been lacking to conduct statistical tests to test this hypothesis. Using our dataset, we were able to determine that Traceability Scores exhibit statistically significant correlations to the following outcomes and rejected the null hypothesis that these correlations were purely random.

1. Faster time to market

The first three tests focus on how Traceability Scores impact cycle time. Do higher Traceability Scores lead to faster test case execution and defect identification? This is a fundamental value asserted by systems engineering and the V-Model – that earlier detection of defects leads to fewer delays and much lower cost to correct. We measured the following times below and noted performance improvements in top versus bottom performers of 2.1X to 5.3X. Higher Traceability scores were found to lead to faster test case execution and defect detection having passed both of our statistical tests.

1. Median Time to Execute Test Cases (2.6X faster)
2. Median Time from Test Start to Defect Detection (5.3X faster)
3. Median Time to Identify the Set of Defects (2.1X faster)

2. Higher quality

The last three tests focus on how Traceability Scores impact quality. Do higher Traceability Scores lead to a higher quality product? This is a fundamental value asserted by systems engineering and the V-Model – that a commitment to test case creation and execution leads to a higher degree of requirement verification and product quality. We measured the following aspects of testing and verification below and noted performance improvements in top versus bottom performers of 1.9X to 2.9X. Higher Traceability scores, having passed both of our statistical tests, led to more tests being completed and a higher percentage of passed tests.

1. Percent of Requirements with Verification Coverage (1.9X higher)
2. Percent of Requirements Verified (2.1X higher)
3. Initial Test Case Failure Rate (2.4X lower)
4. Final Test Case Failure Rate (2.9X lower)

Conclusion

The CMMI defines its best practices in terms of goals, practices, and artifacts. The CMMI does not address the underlying systems and data architecture required to enable these practices, deliver these artifacts, and achieve these goals. The systems architecture reality for most engineering organizations is highly fragmented with the necessary data to manage the engineering product and process (user needs, system level requirements, approvals, component level requirements, model designs, component requirement decompositions, interface definitions, test cases, test results, risk analysis, validations, traceability analysis, etc.) spread across hundreds of siloed tools, spreadsheets, emails, and chat tools with high degrees of uncertainty that any information reflects the latest version continually updated with all interdependencies.

As we have shown, it is extremely challenging if not impossible to move up the CMMI maturity model without addressing the underlying systems architecture and data model. Carnegie Mellon has chosen to use our software to train their students and leading companies have deployed Jama Connect in the ways noted above to achieve their CMMI objectives.

For those interested in exploring this topic further, we encourage you to reach out and have a conversation with one of our experts

Sources:

https://www.cmmi.co.uk/cmmi/cmmi.html
https://resources.jamasoftware.com/whitepaper/requirements-traceability-benchmark

This has been part two of a two-part blog series overviewing our recent whitepaper, “How to Achieve Higher Levels of the Capability Maturity Model Integration (CMMI) with Live Traceability™” Click HERE to read the entire thing.

How to Achieve Higher Levels of the Capability Maturity Model Integration (CMMI): Part 1

The Capability Maturity Model Integration (CMMI), developed at Carnegie Mellon University’s Software Engineering Institute, is a recognized standard for engineering best practices that reduce the risk of defects, delays, cost overruns, and recalls. Organizations that choose to adopt CMMI strive to progress up the five levels in the maturity model by implementing sequentially more advanced best practices spanning the engineering development process.

Jama Software^® is honored to be chosen by Carnegie Mellon as the primary tool used to in its Master of Science in Software Engineering to train the next generation of software engineering leaders in best practices for requirements management, reviews, verification, validation, and process performance management.

The CMMI defines its best practices in terms of goals, practices, and artifacts. The CMMI does not address the underlying systems and data architecture required to enable these practices, deliver these artifacts, and achieve these goals. The systems architecture reality for most engineering organizations is highly fragmented with the necessary data to manage the engineering product and process (user needs, system level requirements, approvals, component level requirements, model designs, component requirement decompositions, interface definitions, test cases, test results, risk analysis, validations, traceability analysis, etc.) spread across hundreds of siloed tools, spreadsheets, emails, and chat tools with high degrees of uncertainty that any information reflects the latest version continually updated with all interdependencies.

The main reason for this landscape of siloed tools is that each engineering discipline is empowered to choose a best-of-breed tool to optimize engineer productivity within their team. The breadth of functionality covered in total by all of these tools — spanning all engineering disciplines — precludes the potential for a single software vendor to provide one software tool which could replace all these best-of-breed tools to the satisfaction of every engineer across disciplines. Generally speaking, each engineering field uses their chosen best-in-class technology to accomplish their objectives. That said, the data needed to achieve CMMI goals, practices, and artifacts is unstructured, unrelated, unconnected, and unmeasurable, which poses a serious challenge when it comes to achieving goals, practices, and artifacts that must span multiple disciplines to control, manage, and improve the engineering process. In order to advance along the maturity model, each engineering organization (regardless of size) needs a unified data model architecture and automated synchronization spanning best-of-breed tools. Without these improvements, most engineering organizations struggle to achieve Level 2 (Managed) and can only do so in a highly manual, after-the-fact manner that generally fails to deliver the desired outcome benefits.

Let’s take a look at a few specific examples from CMMI to demonstrate the need for a unifying data model and an overview of how to achieve it. The first one we will examine is a core practice from the Requirements Management section for Level 2 (Managed) that specifies bidirectional traceability from high level requirements through decomposed requirements and work products across engineering disciplines to generate and maintain a traceability matrix.

CMMI Level 2 (Managed) Requirements Management

RELATED: Tracing Your Way to Success: The Crucial Role of Traceability in Modern Product and Systems Development

There are two ways companies can approach achieving this traceability practice: after-the-fact traceability or Live Traceability™.

• After-the-fact traceability occurs after the product has been developed and is typically a highly manual effort to try and re-create artifacts to demonstrate traceability that should have occurred during the development process but did not. This effort is undertaken solely to comply with industry standards and satisfy auditor requests for demonstration of process maturity.
• Live Traceability occurs in real time as the product development process progresses to improve overall productivity (by ensuring engineers across disciplines are always working off the most recent and correct versions) and to reduce the risk of negative product outcomes (delays, defects, rework, cost overruns, recalls, etc.) through early detection of issues. The benefits of early detection of issues are significant. Research by INCOSE found that issues not found until verification and validation are 40 to 110 times more costly than if found during design. For this reason, most companies want Live Traceability but are stuck with legacy tools and spreadsheets that do not support it. Since each engineering discipline is allowed to choose its own tooling, the result is a large number of tools with no relationship rules or mechanisms to create Live Traceability across them.

So how do you achieve Live Traceability?

STEP 1: Define a Traceability Model

Live Traceability requires a model of the key process elements and their relationship rules to monitor during the development process. Below you see a sample relationship rule diagram from Jama Connect® that defines a common data model that spans best-of-breed tools which enables engineering organizations to manage traceability in real-time and improve process performance. Relationship rules vary by industry and company-specific requirements. Best practice templates are provided to comply with industry standards and configured to meet client-specific needs. The definition of a traceability model forms the foundation for model-based systems engineering (MBSE) since it defines model elements and their relationship to each other in a consistent manner across the entire system architecture.

Step 2: Setup Continuous Sync for Siloed Tools/Spreadsheets

Once the relationship rules are defined, the next step is to set up continuous sync with best-of-breed tools and spreadsheets used by the various engineering disciplines. The traceability diagram below shows a typical example of best-of-breed tools and where they sync in the Jama Connect relationship model to deliver Live Traceability.

Most companies prioritize the areas of the traceability model that are most prone to lead to costly issues in the absence of a continuous sync. Most commonly, these areas are:

• Software task management – directly linking the decomposition of requirements into user stories enables Live Traceability through the software development process through testing and defect management.
• Test automation – test cases are managed in Jama Connect to align to requirements and ensure traceability across all engineering disciplines with the test automation results sync’d to the traceability model at the verification step.
• Risk analysis (DFMEA/FMEA) – is most often conducted in multiple Microsoft Excel spreadsheets and the assumption has been that Live Traceability was not possible with Excel. Jama Connect is the first requirements management solution to enable Live Traceability with Excel functions and spreadsheets. Risk teams can now work in their preferred spreadsheets AND for the first time achieve live traceability to stay in sync with changes made by any engineering team.
• Model-based systems engineering (MBSE) – the first step in MBSE is to define a relationship model between all product requirements. Once a relationship model is defined, then specifications can be determined through modeling. Jama Connect uniquely provides model-based requirements to sync logically with a SysML modeling tool like Cameo No Magic.

RELATED: Traceability Matrix 101: Why It’s Not the Ultimate Solution for Managing Requirements

Step 3: Monitor for Exceptions

Live Traceability provides the ability, for the first time, to manage by exception the end-to-end product development process across all engineering disciplines. The traceability model defines expected process behavior that can be compared to actual activity to generate exceptions. These exceptions are the early warning indicators of issues that most often lead to delays, cost overruns, rework, defects, and recalls. Below is a sample exception management dashboard in Jama Connect.

This has been part one of a two-part blog series overviewing our recent whitepaper, “How to Achieve Higher Levels of the Capability Maturity Model Integration (CMMI) with Live Traceability™” Stay tuned for part two and click HERE to read the entire thing.

Trace Score™ – An Empirical Way to Reduce the Risk of Late Requirements

Executive Summary

One of the main causes of rework, delays, and cost overruns in product development is the creation of new requirements late in the process. This is a well-known risk in product development, but what management practices can empirically be shown to reduce this known risk?

Using our proprietary database of metadata from over 50,000 complex product development projects, we were able to determine that the Trace Score is an empirical method to reduce late requirements. In fact, teams that maintain a high Trace Score reduce the burden late requirements have on their project by 67% compared to teams with low trace scores.

• With this knowledge, our recommendation is that practitioners measure and monitor the Trace Score of their projects to resolve issues early and ensure that the risk of late requirements is kept to a minimum.

Dataset Background

Jama Software has the world’s largest, live dataset of engineering process performance with over 50,000 engineering projects updated and growing continuously. Leveraging this dataset, it is now possible to determine empirically which management practices improve the performance of the product development process. To learn more about our benchmarking, please review our Traceability Benchmarking Report.

The Empirical Questions

In this analysis we will explore three key questions:

1. What are late requirements?
2. How do late requirements negatively impact projects?
3. Does maintaining a high Trace Score reduce the risk of late requirements?

What are late requirements?

For the purpose of this analysis, we define “late requirements” as those requirements created after the completion of a project’s requirement decomposition phase which we estimate as spanning the middle 50% of all requirement creation activity (creation and refinement). To illustrate what late requirements look like, we show two actual projects below with requirement activity plotted over time.

Requirement Creation Over Time

In the Timely Project, requirement creation occurs in a defined requirement decomposition phase to form a necessary and sufficient set of requirements, with very few requirements being added after the fact (e.g. in fig (a), only 1.3% of requirements created late). In the Late Project’, requirement creation bleeds into future phases of the project, leading to a significant amount of late requirements (e.g. in fig (b), 9.2% of requirements are created late).

RELATED: Requirements Traceability Benchmark

How do late requirements negatively impact projects?

We can measure the outsized burden late requirements have on project teams, which we have illustrated for our two projects below. We define late requirement burden as the total number of requirement activities (creation and refinement) attributed to late requirements as a percentage of all requirement activity.

Impact of Late Requirements on Project Team Activity Burden

In the Timely Project, minimal late requirements enable better forecasting of project completion and limit the rework and cost brought on by late requirements (e.g. in fig (c), late requirements only create an additional 8% burden).

In the Late Project, the high volume of late requirements makes it much harder to forecast project completion as the scope of the project is constantly changing, and project teams need to accommodate the late requirements (e.g. in fig (d), late requirements contribute an additional 31% burden).

Unsurprisingly, this additional burden of late requirements has an impact during testing for requirement validation. In our actual project examples, the Late Project has a test failure rate over 3x that of the Timely Project.

RELATED: Unlocking The Power of Live Traceability with Jama Connect

Does maintaining a high Trace Score reduce the risk of late requirements?

A core theorem of Systems Engineering is that maintaining high requirement traceability from the start of a project reduces the risk of late requirements and negative product outcomes. With our project dataset we can now test this theorem empirically. We define traceability as a measure of a project’s ‘expected’ traceability that has actually been established and calculate the Trace Score as follows:(1)

For our example projects, the Timely Project achieved a Trace Score over 6X that of the Late Project; suggesting that maintaining a high Trace Score throughout the project reduces the risk of late requirements.

To further determine if Trace Score correlates to late requirements, we divided our dataset of projects into quartiles based on their Trace Scores (Quartile 1 = bottom 25% trace score, Quartile 4 = top 25% trace score) and then compared the distribution of ‘Late Requirements Burden’ across these quartile groups. What we found is that projects within the bottom traceability quartile had a median Late Requirements Burden 3x greater than those in the top traceability quartile. In other words, the evidence supports that projects managed with higher traceability generally experience less risk from late requirements.

Recommendation

Our analysis has shown that late requirements negatively impact projects and that managing projects through a Trace Score is the only empirical way to reduce the risk of late requirements. Below you can see how one can measure the Trace Score over time as a project progresses to ensure system engineering best practices are being followed. A low or falling Trace Score can quickly identify areas to address to reduce the risk of late requirements.

Here you can see how managing the Trace Score directly as the project is underway would have identified the risk early in the Late Project.

To learn more about achieving Live Traceability™ on your projects, please reach out for a consultation.

Interested in learning more? Download the entire Research Notes: Trace Score datasheet HERE.

Systems Engineers Career Path

Most Systems Engineers we speak with have a common perspective on their role within their organization. Systems engineering as a concept is understood and supported by leadership. And, if systems engineering best practices are followed across all engineering disciplines, leadership also acknowledges the benefits the organization can realize. However, leadership will not force change on engineering disciplines (software, hardware, electrical, risk, verification, and validation) to remove barriers to systems engineering best practice adoption. This leaves most Systems Engineers in the challenging position of possessing responsibility for achieving systems engineering benefits without the authority to ensure engineers adopt best practices. 

The good news is that there is a way out of this situation. The first step to elevating your career is to realize that managing the product engineering process through data is the key and that the Systems Engineering role is the natural role to lead this. For most organizations, the product engineering process is the only process in the company that is not managed through data. No one can go to a system and see the status of all product requirements and where development, integration, risk, and testing stand at any point in time. There is no way to manage by exception. There are no alerts that requirements are not covered, test cases are missed or that hardware made a change that now impacts software. Most organizations have come to accept this state of affairs and try to manage the process through endless meetings, emails, and Slack messages.   

Until the product engineering process is manageable through data, Systems Engineers will be stuck in their current trap — endlessly trying to address issues after the fact, holding unwanted meetings, and the uphill battle of trying to persuade changes in behavior. Below is a 3-step approach that Systems Engineers we work with have used to solve this organizational challenge and have thereby elevated their careers. As with any process change, it is best to do it in stages. You can start with the following steps. 

1. Baseline current process performance 
2. Build the business case for change to gain support 
3. Deliver quick wins

RELATED POST: How to Overcome Organizational Barriers to Live Requirements Traceability

Step One | Baseline Current Traceability Performance 

The first step towards moving the organization to manage the product engineering process through data is to baseline current process performance.  The best place to focus the baseline effort is on traceability since it spans the entire product development process, is a data management concept that is understood, enables systems engineering benefits, and is required by industry standards. To ease the baselining effort, we’ve developed a Traceability Diagnostic that you can use to assess your current situation. The Diagnostic inventories traceable data, the systems in which they reside, and your current Traceability Score™. This is a few-hour effort and forms the basis of the business case in step two.

In this no-cost, guided process, we’ll help you:

• Understand the monetary risk of your current Traceability Debt™
• Uncover the quantifiable ROI of moving to Live Traceability
• Develop a clear plan of action, cost, and timing to achieve Live Traceability

To Get Started With Your Free Traceability Diagnostic,  CLICK HERE

Step Two | Build Business Case for Live Traceability™ 

Once you have established a baseline, it is now possible to build a business case for change that will resonate with leadership. Based on your baseline, the Traceability Diagnostic determines the probability of negative product outcomes (defects, delays, rework, cost overruns, etc.) and the magnitude of these events. This quantifies the risk of maintaining the status quo and doing nothing. In addition to the risk reduction potential of Live Traceability, the Diagnostic also calculates the engineering productivity gains from eliminating the need for time-consuming, manual, after-the-fact traceability efforts. 

Step Three | Start with Quick Wins 

Once you have secured support to move forward, it is common to be able to deliver some quick wins to the organization shortly after project kick-off.  The typical place to start is the painful and time-consuming after-the-fact traceability efforts.  For example, continuous syncing between requirements and software development task management in Jira or Azure DevOps can be set up quickly to automate traceability from requirements to user stories, eliminating a large source of risk and manual, after-the-fact traceability effort. Once quick wins are shown, organizational momentum increases even further and puts you on the success path to begin managing the product engineering process through data.     

Clients of ours that have taken this approach have received significant recognition, been promoted into roles with greater leadership, and have increased their external visibility through speaking engagements. Live Traceability is a unique opportunity to elevate one’s career. Don’t miss the chance. 

Live Requirements Traceability

Achieving Live Traceability™ of product requirements, as necessitated by industry standards, across siloed engineering teams and tools, is the #1 unsolved problem for most product development organizations. One of the main barriers is that each engineering discipline (systems, software, hardware, electrical, risk, verification and validation) has optimized its own process and tools. When looking at the end-to-end product development process siloed teams, tools, and data make it very challenging to trace development activity from initial requirement definition through development and testing. 

As a result, requirements traceability becomes a time-consuming, error-prone, frustrating, and manual, after-the-fact process. The inability for the product development organization to continually trace ongoing development efforts and changes back to user and system requirements results in missed requirements, defects, rework, delays, audit letters, and cost overruns. 

RECOMMENDED READING: Requirements Traceability: How to Go Live

No common platform exists 

The typical approach to solve this generic process problem with software is to force every user onto a single platform and follow one common process. This works for standard business processes in HR, Sales, and Finance, but engineering disciplines across systems, software, hardware, electrical, risk, test, verification, and validation each follow different methodologies and use multiple tools including spreadsheets, desktop, and homegrown applications. Each engineering discipline has optimized their own development environment and strongly resist any attempts to change. Engineering leadership defers to each engineering discipline to define how to best do their work and is loath to dictate processes and tools that will negatively impact the performance and morale of each engineering team.  

In addition to the organizational barriers to standardization, no single platform is even close to currently existing which replaces these dozens of tools. A single platform would need to cover all of the following software categories AND address all functionality in spreadsheets (Excel), scripts, desktop, and homegrown tools: Requirements Management, CAD, MBSE, DFMEA/FMEA, software task management, software code management, automated software testing, hardware bench test tools, ALM, PLM, and more. Current efforts by legacy vendors to create a common SaaS platform to span all these software categories and reach parity with best-of-breed tools is moving very slowly.   

RECOMMENDED READING: Benefits of End-to-End Traceability

How to achieve Live Traceability™ without forcing change on engineering teams 

So how does an organization achieve Live Traceability across a best-of-breed tool environment supporting disparate methodologies, terminologies, fields, and statuses? The answer is a 3-step approach: 

Step One | Live Traceability Model 

Define a Live Traceability model across the end-to-end product development process with relationship rules for the traceable data elements across best-of-breed tools. An automotive functional safety example is shown below. Here you can see the operational instantiation of functional safety standards requirements in a relationship model within Jama Connect®. All necessary traceable information is included with continuous syncs to best-of-breed tools within engineering teams to deliver Live Traceability.  

Step Two | Adaptive Data Field Mapping 

To achieve Live Traceability, integrations with best-of-breed tools (such as those shown in the example) are required. The typical integration approach standardizes field names and statuses to ensure consistency across the connected tool, but this does not achieve the dual objective of Live Traceability with no changes required in how each engineering team works. Alternately, the proven approach is to apply adaptive data field mapping to ensure no change to engineering teams’ fields and statuses which simultaneously ensures a consistent, process wide, Live Traceability model. This is achieved through robust mapping and normalization logic functionality to easily address the various approaches taken by each engineering team. 

RECOMMENDED READING: Requirements Management Guide: Requirements Traceability

Step Three | Management by Exception 

Once Live Traceability is achieved, engineering organizations can — for the first time — manage the end-to-end product development process in real time, identify exceptions to the process early, and take action to significantly reduce defects, rework, delays, and cost overruns.  

LEARN MORE

OKRs (Objectives and Key Results) are a wonderfully simple business tool to explicitly declare what we should focus on with a goal to determine if we succeed. But two critical questions that should be answered before any effort begins, often remain unexamined or not explicitly defined: 

• Why are we focused on this objective and not something else? 
• How are we going to achieve the goal? 

No part of the OKR framework asks these questions and so many companies launch significant efforts against objectives that are inferior and with no defined hypotheses on how to achieve the specified goals. This leads to misallocation of resources and underperformance. In addition, the learning that comes from testing hypotheses is lost so there is no organizational foundation of knowledge to share and build the organizational capacity for growth.     

At Jama Software, we are attempting to address this missing link by adding hypotheses to the OKR framework. Here is a brief overview of our four-step approach: 

Step 1: Define a Business Hypothesis and Train All Employees 

STEM majors are familiar with the scientific method, but not in a business context and if it is not part of company culture it is not often applied. Liberal arts majors are not as familiar with the scientific method and need more training on how to apply it.   

We created a defined template to help jumpstart the development of hypotheses in the OKR framework. The key elements are as follows: 

• Observation – something that makes you go “hmm” and inspires a hypothesis to improve performance 
• Hypothesis – an If/then statement describing an action that leads to an impactful outcome if true 
• Key Metric – what is the one business metric that will be impacted 
• Potential Impact – what is the current level of performance, what higher level of performance is likely if the hypothesis is true? 
• Test – how can the hypothesis be tested quickly and efficiently?  Test results and conclusions are then captured in the system 

As we rolled this out, IDEA became a four–letter word. The reason for this is the following: 

• Ideas often include emotional attachment.  “My idea” vs. “Your idea” where only one can win. A decision about which idea to choose often happens based on hierarchical authority or organizational politics. Not the ideal approach for the best thinking to drive the company forward. 
• Ideas are often little more than a fragmented beginning. Ideas often phrased as “Why don’t we do X?” were commonplace until we rolled out hypothesis training. It became clear to all that this question was a starting point, but not sufficient to raise to others in the organization to engage with. These ideas are now converted into hypotheses that can be built upon.     

Creating a New Hypothesis in Jama Connect

Step 2: Encourage the Creation of Hypotheses 

Anyone at Jama Software can create a hypothesis and share it for input and refinement. This provides a wonderful mechanism for full participation in a learning organization and performance improvement ideas that span functional groups. The structure of the hypothesis makes it very efficient for others to quickly understand the hypothesis and provide refinements through threaded comments. Learning is now structured, shared, and built upon — rather than lost in fragmented thoughts via email, Slack, PowerPoint, etc.     

Step 3: Connect Hypotheses and OKRs 

Every OKR must be linked to at least one hypothesis – the HOW to achieve the key result. We found that initially, for some OKRs, activities could be described, but no one could articulate an actual hypothesis. This made it clear which OKRs were least likely to achieve their key results and resulted in some deeper thinking to come up with an actual hypothesis to test.     

Step 4: Live and Reinforce Hypotheses 

Management must have the discipline to follow the scientific method and reinforce its use across the organization.  It is easy to fall into past bad habits and shortchange the thinking required to articulate a hypothesis.  It most often happens when your belief in your idea is strong and you are emotionally committed to it.  Of course, this is exactly what a hypothesis can address and ensure that the whole team sees its leaders embracing a true learning organization. 

Now that you’ve set your OKRs and connected them to hypotheses, download our free eBook to learn how strategic and structured team collaboration can optimize product development.

DOWNLOAD NOW