Our Test Management Journey: From Jira Xray to GitLab
Test management solutions are designed to help both quality and delivery teams effectively manage, organize, and track their testing activities. Dashlane’s quality team recently decided to implement Jira plugins. We settled on Xray for test management needs and GitLab for our source code management and continuous integration. We found that Xray worked well with Jira, but it could not integrate well with GitLab, our source code and CI tool. Using tooling outside the development ecosystem led to the perception that "quality" was consistently viewed as the responsibility of others rather than aligning with our principle that "quality is a shared responsibility for all."
Our objective was to enhance the sense of quality ownership among the delivery teams, concurrently fostering the adoption of a shift-left culture. Teams responsible for building our product features and experiences (“delivery teams”) were not in complete control over their quality lifecycle, often relying on different tools, processes, and people to tackle each stage.
In this pursuit, it became evident that Xray was no longer the tool fit for our current needs. Our discussions with Xray revealed that there were no plans/intentions to develop either a standalone solution or a Gitlab plugin akin to Jira. We sought a solution that could seamlessly interface with Gitlab, where teams were already managing their work.
If we were to persist with using Jira and Xray solely for testing, it could potentially evolve into an independent project. This scenario raised two primary concerns:
- Difficulty in engaging all stakeholders due to the ecosystem mismatch with our development practices.
- An added burden of maintaining, updating, and synchronizing different project structures with Gitlab, leading to increased overhead.
Our strategy overview
The team started looking at a centralized solution that would integrate with GitLab and facilitate the test and defect creation (bugs/issues/incidents), storage, documentation, execution, and—more importantly—efficient tracking of their progress. We looked at solutions that would integrate with GitLab and tell us more about the value of those tests. We also wanted to keep the source code of our automated tests closely connected to the test cases.
Our selection process comprised of the following stages.
- Evaluating various third-party test management solutions against our decision drivers.
- Assessing GitLab's capabilities in terms of test management against our decision drivers.
- Comparing the most suitable third-party solution with GitLab's capabilities for test management.
A year has passed since our successful transition from Xray and Jira to Gitlab. The statistics and comparisons provided are grounded in the benchmarking conducted in August 2022.
Test management solutions
We looked into various test management solutions featuring strong integrations with GitLab. Our shortlist included Qase, Testmo, Kualitee, Practitest, TestRail, QA Touch, and Tuskr for further investigation.
We evaluated those solutions based on decision drivers and nice-to-have factors.
Bi-directional bug sync: We needed bi-directional bug sync between GitLab and test management tools. We dropped Practitest as it only provided uni-directional sync.
User support: Both Qase and Kualitee had excellent user support and were very responsive in getting back to our questions via chat, email, or video. At that time, Testmo only had email support, but they were responsive.
Test case usability: Qase stood out as the most user-friendly and intuitive, with Testmo following closely in ease of use from our analysis. Kualitee was excluded from consideration as we felt we could get the most out of this solution if we used it for task and project management alongside test and defect management, which didn’t match our requirements. Additionally, Testrail's user experience was deemed less optimal, with the tool's navigation proving challenging based on the feedback from the team overseeing its research.
Value to cost: Testmo was offering the best value proposition after we analyzed their price structure.
Integration with our testing from CI: This involves updating the test case with the latest test execution status obtained from the continuous integration tool. Most of the tools we explored have this ability to some extent, either with APIs or a dedicated integration.
Performance: We examined the response time of test case management tools to assess their performance. Notably, Testmo exhibited remarkable speed even under load, with the capability to import approximately 390 tests within a mere 3 seconds.
Nice to have factors:
Exploratory testing: At Dashlane, as part of our shift-left strategy, we started diving more deeply into exploratory testing techniques. Testmo had a specific menu and tooling to support the explorer, which we didn’t come across in other tools.
Ease of migration from Jira: We only evaluated a few tools for this factor. Qase was an outperformer as it had dedicated integration support from the Xray plugin for Jira.
We chose Testmo as the finalist because it was a simple and easy-to-use solution that had the best value proposition. It also offered better organization of tests and dedicated support for exploratory testing.
However, having a tool outside the GitLab ecosystem is still a challenge: It makes it harder to involve all stakeholders and track progress made on issues since they live outside GitLab Dashboards.
Using GitLab for test management
GitLab test case
However, even though this feature was integrated, it was still too immature for our needs. The issue type has no assignees, test executions, priorities, linking-related issues, and so on.
We dropped it as it didn’t match our expectations. Nevertheless, it made us take a deeper look into the GitLab issue structure.
GitLab issue type organization for test management
The way we organized our testing in GitLab helped us consider a different approach: We could use GitLab issues as our "Test Management Tool" since it could fit our needs.
How do we organize our testing in Gitlab?
We used GitLab issue type with scoped labels to identify the test cases. To ease test creation and to ensure a consistent format across the organization, we built a test case template to guide the test creation. Preconditions were separate issue types in Xray, but we made it an integrated part of the test cases in GitLab.
One challenge we encountered was associating the issues (tests) on GitLab with the execution results of tests on Continuous Integration, even though both exist on the same platform. Additionally, there was no option to associate GitLab’s test cases with its execution at the time of our assessment. This meant that we couldn’t know when tests were actually completed.
With Xray, Testmo, and other test case management solutions, we had the capability to directly link the test execution results from GitLab CI to the issue.
We found a hack for tracking test executions by updating a dedicated label, posted by Mek from GitLab. We used scoped labels to mark the test executions.
While this hack isn’t a better experience than using test management tools, it is a manageable solution for our needs.
We used GitLab issue types with scoped labels and bug templates to guide the bug creation process, similar to what we set up with test cases.
On top of that, we used a Slack bot to streamline reporting bugs.
We were using test plans as a separate issue type in Xray, and we decided not to have a dedicated issue for it in GitLab. Instead, we set up test planning to be part of user stories as acceptance criteria.
Monitoring via quality dashboards
For specific graphs that we can't extract from GitLab, including Automation Score, Manual vs Automation breakdown, and Feature/Platform wise breakdown, we need to rely on some visualization tool for charts by extracting data from GitLab. In our case, we use Tableau/Kibana to visualize. (You can see an example dashboard here.)
This has been one drawback of our transition away from Jira. Xray used to provide on-the-go dashboards on their platform like other test management solutions. But in GitLab, we had to push all the issue data into Tableau/Azure workbooks. The information is there, but additional effort needs to be made to put it in an accessible format.
Displaying Jira issues and their state on Confluence via dynamic Jira filters
Due to the seamless integration between Jira and Confluence, it is possible to showcase Jira issues directly on Confluence pages. This integration ensures that real-time updates regarding issue status, resolution, summary, execution status, automation status, and more are reflected. All someone has to do is paste the JQL (Jira Query language) in Confluence to get a comprehensive view, eliminating the need to navigate to Jira separately. Unfortunately, we don’t have something quite as integrated with GitLab. While the GitLab connector plugin offers the potential for achieving analogous functionalities, it is important to note that it does not replicate the exact same experience.
Our final decision
Ultimately, we decided to use GitLab issues for test management as it’s easier to involve all stakeholders, requires less context switching, is cost-effective, and we didn’t have to significantly compromise on any decision drivers.
Additionally, GitLab is investing more in test management, so we can anticipate many new capabilities being added soon. For example, test sessions are something we’re looking forward to.
Are you or someone you know a skilled Mobile SDET with a passion for quality? Join Dashlane, and play a key role in elevating the quality of our mobile apps. We're seeking individuals with strong mobile automation expertise, preferably skilled with Kotlin.
You choose your base: Lisbon or Paris. We offer relocation assistance and visa support.
Sign up to receive news and updates about Dashlane
Thanks! You're subscribed. Be on the lookout for updates straight to your inbox.
With Dashlane since 2016, Sagar is a Senior Quality Engineering Manager with over a decade of experience in the software industry. He's particularly passionate about Quality, Automation, and DevOps, and he has a Master's of Science in Communication & Computer Security from Telecom ParisTech in France. Currently based in Paris, he hails from Bangalore, the Silicon Valley of India.