Overview

Validation of a new static code analysis tool for AUTOSAR software development.

The project aimed to qualify and validate the tool’s ability to perform static analysis on AUTOSAR-compliant software, ensuring that the analysis results were reliable, relevant, and consistent with the expected software requirements.

I was responsible for the validation activities from end to end, including the definition of the test strategy, design and implementation of test cases, and development of the test tools and framework used to automate the validation process.

Context

The project was introducing a new tool dedicated to the static analysis and verification of AUTOSAR source code.

The tool was intended to support software development teams by automatically detecting coding discrepancies with AUTOSAR requirements, and non-compliant implementation patterns.

Before being shiped to customer, the tool needed to be thoroughly validated to ensure that its results were correct, reproducible, and trustworthy.

The validation required a structured approach covering different types of AUTOSAR code patterns, coding rules, expected violations, and edge cases.

Challenge

The main challenges were:

• Define a comprehensive validation strategy for a new static analysis tool
• Identify relevant test scenarios covering AUTOSAR-specific coding patterns
• Verify both positive and negative detection cases
• Ensure that reported violations matched the expected analysis results
• Detect potential false positives and false negatives
• Build a test environment capable of executing a large number of validation cases
• Automate the execution and evaluation of test results
• Ensure test results were reproducible and traceable
• Handle different source code configurations and analysis rules
• Provide a reliable framework that could be reused for future tool versions
• Validate the tool against the expected behavior before its integration into the development process

My Contribution

I was responsible for the complete validation activity, from test strategy definition to implementation and execution.

My main responsibilities included:

• Defining the overall test and validation strategy
• Identifying validation objectives, test scopes, and acceptance criteria
• Designing test cases covering AUTOSAR coding patterns and analysis rules
• Creating source code samples specifically designed to trigger expected analysis results
• Developing automated test tools and validation utilities
• Designing and implementing the test framework used to execute the validation campaigns
• Implementing automated comparison between expected and actual analysis results
• Developing mechanisms for test result collection, reporting, and traceability
• Investigating discrepancies between expected and observed tool behavior
• Identifying false positives, false negatives, and unexpected analysis behavior
• Executing validation campaigns and analyzing their results
• Reporting defects and inconsistencies identified during the validation
• Maintaining and extending the validation framework as the tool evolved

Solution

A dedicated automated validation framework was developed to systematically evaluate the static analysis tool.

The validation process was based on a set of controlled source code examples representing different AUTOSAR coding situations and expected analysis outcomes.

Each test case defined the expected behavior of the static analysis tool, including whether a specific coding pattern should be detected and which analysis result was expected.

The framework automated the complete validation workflow:

• Prepare the source code associated with the test case
• Execute the static analysis tool
• Collect and parse the generated analysis results
• Compare the actual results with the expected results
• Identify discrepancies and classify validation failures
• Generate test reports and validation results

The test suite included both positive cases, where a violation was intentionally introduced and expected to be detected, and negative cases, where compliant code was used to verify that the tool did not incorrectly report a violation.

This approach made it possible to systematically assess the behavior of the static analysis tool and quickly identify regressions when new versions were released.

Architecture