The main idea behind Sonargraph has always been to provide a tool that eases the creation and maintenance of high-quality software. For any serious project that must live longer than a couple of months, it is actually cheaper to spend a part of your resources to keep your software constantly at a good level of quality than using all your time to create new features. Martin Fowler explains this very well in his article “Is High Quality Worth the Cost?”. The bottom line is, that apart from the very early development stages, high-quality software is actually cheaper to develop, because it allows adding new features at almost constant speed, whereas it becomes more and more time consuming to add new features into a code base with low quality. According to our experience the most successful teams spend about 15% to 20% of their time on code hygiene.
We at hello2morrow believe that a consistent architecture is a fundamental part of software quality. When we use the term “architecture”, we think of it in terms of the IEEE 1471 standard:
“The fundamental organization of a system embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution.”
This blog post describes why architectural design as an activity is needed, why conformance checks need to be done automatically by a tool and how Sonargraph supports you as a developer and architect during these activities.
The Need for Architecture
Martin Fowler wrote another excellent article “Is Design Dead?” back in 2004. He argues that for anything serious, you cannot just code along and hope for the best, but need “planned design”:
“If you want to build a doghouse, you can just get some wood together and get a rough shape. However if you want to build a skyscraper, you can’t work that way – it’ll just collapse before you even get half way up.”
Martin Fowler
He is not arguing to design everything up front, but rather making the design activity part of the agile development process. As a consequence, architecture needs to evolve together with the code base and the knowledge of the team.
If you take the definition for architecture mentioned at the start of the chapter, then the top-level architecture should contain the main components and their dependencies. As with construction architectures for large buildings, no single diagram exists that contains all information for a large-scale software system. There is an upper limit of elements that our brain can process, especially if there are also interconnections of different types between elements. Thus, if the system grows beyond a certain size, abstractions are needed which can be thought of as maps at different scale. Simon Brown gives an example of this with the C4 model. Where details are needed, additional diagrams can be created.
Of course, several viewpoints for software architecture exists as described by “The 4+1 ViewModel”. This “static” architecture that describes the decomposition of a system in its parts is the foundation for the “dynamic” aspects like information flow: If there is no direct dependency between two components or between them and any of their commonly shared components, there cannot be any information flow between them. Consistency of the diagrams now becomes a challenge: Higher-level abstractions must not be violated at the lower level: There must not be a secret tunnel at the detailed level where the higher level puts a clear barrier between components.
The last decades of agile software development have shown that it is impossible and impractical to do a big design upfront. Not only the requirements from the outside (read ‘business’) may change, but most certainly the developers’ understanding of the domain improves over time and thus the architecture likely also needs to be adapted as a consequence. The effort needed to change a functionality and the effects this change causes for the rest of the system very much depend on the number of usages of this functionality.
Design for changeability therefore means to minimize coupling (i.e. the number of dependencies) between elements, because elements with low coupling can be more easily re-arranged. Especially bad for coupling are cyclic dependencies that may cause mental challenges in form of hen-and-egg problems and also make it harder to understand a system’s structure if a large number of elements are involved. You want to avoid an entangled mess as shown in the following picture (where green arcs represent dependencies between Java packages and the direction of dependencies is counter-clockwise) that is often described as “big ball of mud”:
We believe that architectural debt is the most toxic form of technical debt, because fixing architectural issues usually requires wide-spread changes that are quite risky in itself. As a consequence, we think that spending effort on a clean and consistent software architecture and controlling dependencies is essential to code quality as well as regularly cleaning up other code smells. Our experiences match those of well-known experts. Here is an incomplete list of resources that we found inspiring and affirmative to our thinking. They haven’t lost any importance despite some of them dating back a couple of years:
- “Is High Quality Worth the Cost” by Martin Fowler, https://martinfowler.com/articles/is-quality-worth-cost.html, 2019
- “Sustainable Software Architecture: Analyze and Reduce Technical Debt” by Dr Carola Lilienthal, dpunkt.verlag, 2020
- “Is Design Dead” by Martin Fowler, https://martinfowler.com/articles/designDead.html, 2004
- “Large-Scale C++ Software Design” by John Lakos, Addison-Wesley, 1996
- “The Pragmatic Programmer: From Journeyman to Master” by Andrew Hunt and David Thomas, Addison-Wesley, 1999
- “Applying UML And Patterns” by Craig Larman, Prentice Hall, 2000
- “Refactoring to Patterns” by Joshua Kerievsky, Addison-Wesley, 2005
- “Agile Software Development” by Robert C. Martin, Prentice Hall, 2003
Automating Checks
Manually tracking the evolution of the internal structure of a software system is not efficiently manageable. With thousands of classes and millions of dependencies this is impossible for any large system. You need a tool that automatically reports deviations in the implementation from your envisioned architecture. We think that the lack of proper tool support is the reason why so many projects suffer from software rot. There are many static analysis tools and linters that report errors and problems at source-file level, but most are missing out on the big picture: Tracking dependencies across source files and validating if the structure matches the envisioned design.
As a result, most systems contain a large number of cycle groups, elements are tightly coupled, no clear structure exists and if documentation exists, it is not up-to-date. New developers have a hard time to know how they should structure new functionality, where to place the new code and how it should interact with the existing code. Once you have lost control, the structural quality typically spirals downwards quickly and the software ends up in a “big ball of mud”. Regular quality initiatives feel like Sisyphean tasks, because problems are created faster than they can be fixed.
Therefore, quality checks must be executed automatically by the Continuous Integration (CI) build, whenever new code is committed. Even better, quality checks should be executed while programming, so that those problems never get into the version control system.
Setting the Focus
Starting a new project with automated architecture checks in place is very practicable to maintain quality at a high level. It is harder to sustainably improve quality for existing projects. Resources are scarce, and new features need to be implemented, so it is not feasible to simply stop all work and clean up everything first. Instead we highly recommend incremental and measurable improvements. For any quality improvements, it is important to spend efforts where it supports the current work. This makes the positive effects visible, and the enthusiasm for code quality will stay and won’t fade away quickly. For this reason, Sonargraph offers to compare the current state of the system against a previously run analysis. This ‘System Diff’ identifies where quality was improved or worsened, making it ideal to support code and sprint reviews. Additionally, quality gates can be defined that ensure that quality trends in the right direction. Upcoming in the next release (11.5.0) is a ranking algorithm that highlights those issues where fixes provide high benefit, i.e. that are urgent and important: Issues that were added recently, that have a high impact on the system and where involved files have been changed recently.
The Sonargraph Product Family
The rest of the blog post gives a 10000 ft overview of the Sonargraph products. Details of specific functionality and use cases can be found in other blog posts and the user manual.
Sonargraph focusses on architecture and dependencies, but it offers also a large number of metrics, duplicate code checks and additional rules that can be activated as needed, e.g. to detect unused code. Sonargraph is built upon the experiences that hello2morrow gained during the development and support of the predecessor products Sonargraph 7, SotoGraph and SotoArc. Sonargraph is lightweight and integrates smoothly with different IDEs, build and quality infrastructures (e.g. Eclipse, IntelliJ, SonarQube, Jenkins, Ant, Maven, …).
Sonargraph consists of several products that help to ensure quality throughout the software development as shown in the following image:
- Sonargraph-Architect allows code exploration and definition of rules, i.e. architectures, metrics, anti-patterns, thresholds, tasks, refactorings. It offers additional analyzers. e.g. to detect code duplications and to provide custom metrics and issues.
- Sonargraph-Developer are integrations into IDEs that provide early feedback to developers. With a Developer license it is also possible to start the Sonargraph-Architect application in read-only mode and use its advanced visualization and exploration capabilities.
- Sonargraph-Enterprise is a web application that provides the history of metrics for multiple Sonargraph systems.
- Sonargraph-Build are integrations for various environments to run the quality checks headless, e.g. on the continuous integration server.
- Further plugins exist that allow the integration of Sonargraph into SonarQube and Jenkins.
We host an Open Source project on GitHub that provides easy access to all information contained in a Sonargraph XML report and can be used for custom post-processing: https://github.com/sonargraph/sonargraph-integration-access
Use Cases and Key Functionality
The following describes key functionality of Sonargraph and typical uses cases. This is just a summary, the user manual provides more details.
Architecture Definition
Sonargraph uses a Domain Specific Language (DSL) approach to describe the architecture. A system’s architecture can consist of multiple architecture aspects which are checked in parallel. Alternatively, the architecture can be defined interactively. Architecture diagrams can be generated allowing to investigate connections between architecture artifacts, see “Visualizing an Architecture as a UML Component Diagram”.
Simulate Refactorings
Sonargraph allows the simulation of refactorings. For this, you can create multiple so-called virtual models. A virtual model is a space where the model from the parser(s) can be modified by refactorings and detected issues can be transformed into tasks or ignored (called resolutions). This allows the simulation of different approaches to change a given structure.
The ‘cycle-breakup’ analyzer proposes refactorings to find an efficient way to eliminate a cycle. It takes into account defined architectures and allows to interactively fine-tune the solution.
Hotspot Visualization
Sonargraph analyzes information from Source-Control Management (SCM), currently Git. The combination of issues and code changes and the visualization as software maps (a.ka. “Code Cities”) allows the visual identification of hotspots.
Tracking Changes in Quality
If Sonargraph is used in existing projects there might be an overwhelming number of reported issues. The ‘System Diff’ analyzer allows focussing on changes, making it the ideal companion during reviews. Quality gates can be defined on the current system state or in comparison to a baseline, making it easy to follow the ‘Boyscout Rule’ and gradually improving the system’s quality. The ‘Issue Ranking’ view recommends issues that are both urgent and important to fix.
Great Parser Model Detail, Little Memory Consumption
Dependencies are tracked down to method and field level offering detailed exploration. Sonargraph has little memory consumption, as only the model coming from the different parsers is held in memory and all ‘derived’ structural elements (e.g. a layer) and their dependencies are calculated on demand.
Snapshots
The complete model of a system is stored in a compact binary format. This enables fast startup times (the last snapshot is used if available) without having to perform a full re-parse. Furthermore, complete systems might be compared and historically analyzed – even passed around to enable reviews based on them – by directly loading the snapshot.
Fast Execution
Analyzers calculate metrics and analyze dependency structures (e.g. cycles) and content of source files (e.g. duplicated code). These analyzers run in parallel in a multi-threaded environment providing more speed while not blocking user interaction. Once an analyzer has finished, it`s results are available to the user.
Extensible Analysis
The user can extend Sonargraph’s functionality by writing Groovy scripts accessing the model created by Sonargraph. These scripts can either simply act as custom queries finding artifacts with specific characteristics and/or to create issues pointing to potential problems in the system or create additional metrics.
Sonargraph also offers a plugin API to integrate external analyzers and to extend the parser model by custom elements. The currently existing plugins are ‘Spotbugs’ and ‘PMD’ for further file-local issues and ‘Swagger’ and ‘Spring Microservices’ to reveal web service dependencies.
Multiple Language Support
Sonargraph supports different languages depending only on the license without the need to have different installations. There is a unified approach (i.e. one user interface) to explore and monitor systems implemented in different languages. Systems have a module structure where each module can have a different language.
Inter-module dependencies with different languages are detected where possible (e.g. by analyzing JNI calls). A generic component approach is used for all supported languages – currently Java/Kotlin, C#, C/C++, Python.
Flexible Exploration of Dependency Structures
You are free to decide how to explore dependencies. Sonargraph offers a tree-like explorer, a graph viewer and a simple table-based viewer.
Automated Updates and Flexible User Interface
Sonargraph is built upon the Eclipse Rich Client Platform (RCP) and thus prompts you as soon as updates are available. Views can be flexibly arranged and context menu interactions allow smooth navigation. Sonargraph-Build plugins for Maven and Gradle can also be configured to update automatically.
Exchangeable Quality Artifacts
The software system analysis comes with a multiple file approach: The software system is comprised of a main software system file, analyzer configurations, user defined scripts, different architecture aspects and so forth. All files are text-based, so the quality definition can and should be placed into version control together with the source code. The approach makes it easy to track changes and to share valuable aspects of the analysis between software systems as well as to centralize common aspects in bigger companies.
Summary
This blog post illustrated the motivation behind Sonargraph’s development: We think (and others, too!) that dependency management is essential to code quality. It is important that the checks are run automatically and require little to no effort. With powerful visualizations and simulated refactorings Sonargraph enables the efficient analysis and improvement of existing code bases. Our customers report significant speedup of development and we can say from our own experiences that programming a well-structured code base is a lot of fun.
Give it a try! Evaluation licenses allow two weeks free usage of all Sonargraph Architect features.