Overview
The Contract Test Generator is a utility collection that Qodex users can fork into their workspaces to automatically generate potentially hundreds of live tests against their APIs.
Demo
Problem being solved
Most APIs today will ship with some form of a contract or specification. This could be OpenAPI, WSDL, GraphQL or Protobuf. The contract defines exactly how the API should work including endpoints, parameters, and request/response schemas.
API producers who are fulfilling the server side of these contracts do their best to match the contract, but there can sometimes be gaps. A field might be marked as not required, when actually it is. Or a value might be marked as an integer when it's actually returning as a string.
Contract testing is a way that you can validate your implementation of a contract matches the specification that is provided.
This often comes in two forms; consumer-driven contract testing and producer contract testing. This workspace and the utility collections within aim to simplify and improve the producer contract testing flows within Qodex.
What does it do?
This utility collection will:
Introspect your API Definition file (OAS2/OAS3)
Validate the API Definition file meets minimum criteria (linting, style, naming conventions, examples)
Generate a series of tests to execute against each endpoint
Execute the tests against the URL provided
Produce results in the collection runner for users to validate.
This utility collection doesn't:
Generate any subsequent collections that can be used later - all data is processed at runtime.
Produce any visual reports outside of the Qodex API Platform - however you can use newman to do this.
Getting Started
The first step is to create a new workspace in your Qodex team to house this collection along with the API Definition file that you are looking to test.
The workspace should have only 1 API Definition available to ensure the tests run smoothly.
Import the API Definition into your workspace. This should be visible inside the API tab within Qodex.
Fork the appropriate collection from this workspace (OAS2/OAS3) into your new workspace.
Fork the Contract Test Environment environment file from this workspace into your new workspace.
To get this working you need to populate the environment file in your workspace with your specific API parameters.
Configuration
The following section describes the different environment variables and how these should be used.
| Variable Name | Description | What you need to provide |
|---|---|---|
| env-apiKey | Your Qodex API key used to access the Qodex API | Browse to your Qodex profile and generate a new API key. Copy and paste the contents into the current value of this variable. |
| env-minApiCount | The minimum amount of APIs required in a provided workspace | This should be left as 1 as you should only have 1 API in your workspace. |
| env-maxApiCount | The maximum amount of APIs required in a provided workspace | This should be left as 1 as you should only have 1 API in your workspace. |
| env-workspaceId | The identifier of the workspace you wish to generate tests for | You can get this workspace ID by browsing to the Workspace Overview page in your current workspace and clicking the 'information' icon on the top left. This should be a GUID value e.g. 0bc7d76d-b582-45ba-b216-5da2c1d174a4 |
| env-requireParamDescription | Flag denoting if assertions should be run that require a parameter description. | Enter 'true' into this variable if you want the generator to validate whether each of your parameters in your API Definition have a valid 'description' property. |
| env-requireParamExample | Flag denoting if assertions should be run that require a parameter example | Enter 'true' into this variable if you want the generator to validate whether each of your parameters in your API Definition have a valid 'example' property. |
| env-paramDescriptionMinLength | The minimum length a description should be. This is only used if env-requireParamDescription is true | Enter a positive integer into this variable if you want the generator to validate whether each of your parameters in your API Definition have a description property that has length greater than this min length. |
| env-paramDesciptionMaxLength | The maximum length a description should be. This is only used if env-requireParamDescription is true | Enter a positive integer into this variable if you want the generator to validate whether each of your parameters in your API Definition have a description property that has length fewer than this min length. |
| env-securityExtensionName | If using a role-based API, the name of the extension you use to denote allowed roles on an endpoint | This is only valid when using OAuth2.0 with roles and scopes. Further documentation is required to be written about how to get this to work. |
| env-roleHeaderName | If using a role-based API, the name of the header where you supply the user's assumed role | This is only valid when using OAuth2.0 with roles and scopes. Further documentation is required to be written about how to get this to work. |
| env-server | The description of which server element to use. This is for the base url of your API. | You need to populate this with a live server to test the contracts against. This could be localhost, a Qodex Mock Server or a live instance. Important: this url also must match one of the URLs provided in your servers object in your API Specification. |
| env-schemaUrl | The url to the OpenAPI spec that you want to validate against. | You can populate this field instead of supplying an API key and WorkspaceId. The generator will instead pull the spec from this URL. |
| env-apiIds | A comma separated list of Qodex API IDs that you would like to validate in the supplied workspaceId. | You can use this field if you optionally want to validate only a subset of APIs within a particular workspace. |
| env-runComponentTests | Flag denoting if assertions should be run validating schema adherence | Should be set to 'true'. This is a flag that will tell the generator to run the linting and style guide checks against your API Specification. |
| env-runContractTests | Flag denoting if contract tests should be generated and run | Should be set to 'true'. This is a flag that will tell the generator to run the automated contract tests against the server specified in env-server. |
| env-schemaPropertyExceptions | The names of any properties that should not be validated in the component tests | This should be populated with an array of strings to specify any properties that should be skipped from the validation step. |
| env-jsonToYaml | NPM Package that converts yaml to json. | DO NOT EDIT THIS VARIABLE. It is required for the processing of the contract tests. |
Running the Contract Test Generator
Once you have populated the environment variables on your workspace then you can go ahead and run this as a collection. Make sure you set the environment to utilise the Contract Test Environment or you will get errors.
Support
This collection is maintained in GitHub at https://github.com/Qodex-solutions-eng/Qodex-contract-test-generator. For support please post issues directly in that repository.
Acknowledgements
The initial version of this collection was created by @allenheltondev. Acknowledment must go to him and the team that worked together to create this tool for the benefit of others.