Paylocity QA teams drive test automation with Postman

A few years ago, the Paylocity platform was primarily a monolithic application. Monoliths are notoriously challenging to build, test, and deploy, and—perhaps more importantly for Paylocity—they're also problematic for any organization that wants to deliver new releases quickly. The engineering organization decided to migrate to a microservices architecture to help them deliver new releases faster. Now, each engineering team is responsible for delivering a particular service, many of which include one or more APIs.

In 2015, with this growing number of microservices—related APIs, Paylocity's Expense team adopted Postman to help them manage API test operations. Inspired by their positive experience with Postman, the team evangelized the platform internally and built test components on the platform that could be used by other teams. Due to Postman's ease of use, developers and engineers were able to dive right in.

“When I first tried Postman, I found it to be so much easier to use than anything I've ever used before. I got started right away and have never looked back.”

Kristin Jackvony, QA Lead, API Services, Paylocity

Since then, Postman usage has spread across the Paylocity engineering organization with more and more QA engineers now using Postman in their daily workflows.

“Postman has helped me expand my understanding of how APIs work. When I first tried Postman, I was blown away. I wish I had discovered it years ago!”

Austin Mills, QA Engineer, Cash Management Team, Paylocity

Automating a broad range of QA use cases

As more Paylocity QA engineers and teams began using Postman, they developed new ways to leverage the platform to simplify and automate testing and other tasks, meaning faster delivery of impactful enhancements to clients. The following are some of the more prominent examples.

• Self-service API and data reference

  Throughout the development cycle, developers and QA engineers need an easy way to access API information and run requests. Paylocity's API Services team built a shared Postman Collection that includes requests for all the APIs they manage. This enables the team to quickly find what they need, whenever they need it. Developers can go in and check API functionality and QA engineers can easily put together requests for manual testing.

  Some Paylocity teams, like the Application Security team, need test data during development. In the past, this would involve a time-consuming workflow on the QA side to set this up. Now, a QA engineer can simply send over the relevant Postman Collection and environment variables, and the developers can generate whatever data they need themselves.

• Environment monitoring

  Many QA teams at Paylocity run daily diagnostics on their environments to make sure they are running as expected. The API Services team automated this process on Postman using the Newman command line Collection Runner, as well as custom webhooks to integrate with CI/CD tools. Each night, a PowerShell script triggers Newman to begin the tests.

• Smoke testing

  Newman has proven useful in a wide range of automations for Paylocity. The API Services team, for example, uses Newman to conduct automated smoke tests in their deployment system. They've configured Newman to run certain requests to determine whether or not a build is stable enough for more exhaustive testing. With smoke test automation set up, no matter which environment the team deploys to, Newman will kick in and run a smoke test.

• Test setup

  Paylocity QA teams use scripts and variables in Postman to help them automate many time-consuming manual workflows. Scripts allow QA engineers to write test suites that include dynamic behavior in requests and collections. Variables allow teams to store and reuse variables in requests and scripts, bypassing the need to manually enter data in every place that it's needed. Global variables (one of Postman's variable scopes) have been particularly helpful in enabling teams to reuse data between collections, requests, scripts, and environments.

  In one example, the External Integrations team wrote a lengthy script to automate the detailed process of setting up a new customer record. Each time they run certain tests, the script creates a company record, sets configuration settings, and adds the required data.

• Authentication testing

  Postman helps Paylocity automate many repetitive tasks, such as generating test data. The Year End team wrote a complex script that enables QA engineers to grab tokens during user authentication testing. The script triggers a JSON data file to feed into the collection runner that, in turn, injects different credentials to the token service. Depending on user permissions, a particular authentication token is returned. The QA engineer uses that token to test that the API returns the correct user status, ensuring that users with certain credentials can't hit the API.

• Sensitive data handling

  As part of the Paylocity service, some APIs handle sensitive data, such as employee social security numbers, salaries, and the like. Data security is paramount, and the company ensures that all such data is masked for internal development and testing. In addition, some APIs are customer-facing, and a customer's data must only be available to authorized individuals at that company.

  The Cash Management team wrote a test script to ensure that sensitive data is accessed correctly. The script invokes Newman to run collections that cycle through numerous use cases and user roles associated with internal and customer-facing APIs. This process involves a range of test permutations that would have required significant time and effort to accomplish manually.

• Performance benchmarking

  The Position Management team also conducts a lot of data-driven testing on Postman. To benchmark API performance, the QA engineers created a collection that hits the same endpoints across a large number of companies in their test environments. The response times helped the team create a baseline that can now inform a variety of projects, such as the impact of a Microsoft SQL Server upgrade.

• Load testing

  Occasionally, Paylocity developers need to simulate real-world scenarios in order to test API performance. Postman enables QA engineers to quickly set up and run simple automations to accomplish such tasks. For example, when a developer needed to load test an email service, the API Services team built a collection that would trigger an email. They used Newman to run the collection a hundred times in succession, generating enough load to inform the developer's work.

• Third-party service integration testing

  Paylocity integrates third-party cloud services that power special platform functionality. To help test their RabbitMQ integration, the Cash Management team built a collection that sends requests to the service to GET or POST a message on the various nodes that they have set up in RabbitMQ. This enables the team to quickly see which nodes are up and running normally and which need troubleshooting.

• Data validation

  Some services at Paylocity use a nested data structure. When the service needs to toggle between data sets during an API call, some lower-level data may get missed and the call returns an incomplete result. Previously, the workaround was to run some SQL to fix the issue. With Postman, Paylocity's QA engineers wrote a script that feeds the various layers of data into a file before running a request. When a result is returned, the team can see straight away if any data set got missed. This was a major win for the QA team. Out of 1,800 requests, they were able to catch a few instances of missed data, which would have been nearly impossible with manual spot testing. The automated test not only saved time but also prevented new issues arising in the future.

• Recruiting assessments

  During the QA engineer hiring process, Paylocity tests candidates on their knowledge and skills. The company wanted to make sure their new hires had a good understanding of APIs and how to build a quality test plan. Postman became part of the assessment in the form of an example collection with associated documentation.

The focus on test automation at Paylocity is paying off. Using Postman, QA engineers are able to turn around test results much faster, and the team's velocity has increased. This enables QA and developers to identify and troubleshoot issues more quickly, which speeds up the development cycle as a whole.

“One of the first things I do every morning is look at the automated Postman tests that run overnight in our CI server using Newman. When the developers start their day, they'll know right away if something needs attention. It's a huge time-saver for our team.”

Austin Mills, QA Engineer, Cash Management Team, Paylocity

For many test cases, the QA engineer is able to get the automation set up in Postman before a new build is even ready for testing.

“Postman enables me to quickly build out a test automation and hit the ground running. I'd say 90% of my API test cases are run on Postman.”

Annamalai Viswanathan, QA Engineer, Position Management Team, Paylocity

Test automation also results in more time for QA engineers to focus on other things. For example, they're able to do more exploratory testing.

“With our automations on Postman, I don't have to worry about things like API regression testing. I can deep-dive into a new feature and come up with creative ways to test it.”

Kristin Jackvony, QA Lead, API Services, Paylocity

When they need help, Paylocity teams turn to the Postman user community through channels including Postman forums, Stack Overflow, GitHub, and more. Tips, ideas, and advice from other engineers can help them quickly optimize testing or build new automations.

Additional usage of Postman is already in the works: Going forward, Paylocity teams plan to leverage Postman's API documentation generator and Swagger support to improve their approach to documentation, as well as Postman's API Visualizer to help make results data easier to share. Ultimately, the more they play around with Postman features, the more they can streamline their own workflows and share their learnings with other teams.

“I'm learning new things on Postman all the time, and it's one of the most fun parts of my job. I feel like a kid in a candy store.”

Annamalai Viswanathan, QA Engineer, Position Management Team, Paylocity