December 18, 2024

NestJS integration testing with Testcontainers

Goal of this article

The goal of this article is to present a solution for implementing integration tests in a Nest.js application. Specifically, we aim to emphasise good practices for creating a testing environment and mocking external dependencies of the tested application. It’s also worth mentioning that the provided approach will use Testcontainers for databases instead of traditional mocking.

Whats integration testing

Unit testing focuses on testing individual components of the software, mocking all external dependencies and focusing on local logic. Integration testing goes one step further. While unit tests ensure that each component works correctly in isolation, integration tests validate how these components work together.

For example, even if all components pass their unit tests, they may still have issues interacting with one another. Integration tests verify entire functionalities, such as specific endpoints, along with their underlying business logic. It is important to note that while integration tests focus on local business logic, some external dependencies still need to be mocked. Specifically, external API calls should be mocked to ensure tests remain focused on the application's core functionality.

Integration testing in Nest.js

The most important goal of an integration test environment is to mimic a clean production environment as closely as possible. We will achieve this by using containerization of services.

Nest.js integrates two excellent testing libraries that we will use extensively in our project:

Jest

A general-purpose testing library that provides an interface for writing and executing test suites.

Supertest

A library for testing web applications and API services. It simplifies placing API requests and verifying server responses.

In addition to these, for containerization, we will use:

Testcontainers

A library that simplifies the creation and management of Docker containers for integration tests. It provides programmatic access to containers, allowing us to easily spin up and manage services like PostgreSQL and Redis during our tests. For this example, we will deploy two Testcontainers: one for PostgreSQL and one for Redis. These containers will be reset after each test case to ensure a clean testing environment.

Nest.js offers seamless integration with Jest and Supertest right out of the box.

Example project

The example project for this article simulates partial logic for accepting and processing online shop orders. Its main goal is to demonstrate an app that uses PostgreSQL and Redis databases while heavily relying on external services. In this case, we use an API from fakeApiStore.com to fetch data during processing.

Project Architecture

The project architecture consists of:

- Nest.js application server
- PostgreSQL server
- Redis server

Workflow

The Nest.js application processes a request to finalise a specific cart. Finalisation involves the following steps:

1. Fetching cart data from an external API
2. Calculating the total value of the order
- This involves fetching data for each individual product in the cart from an external API or retrieving it from Redis Cache.
3. Creating a Customer entity (if not already present in PostgreSQL)
4. Saving the processed Order in PostgreSQL

This structure allows us to highlight several critical considerations when setting up and implementing integration tests in your project.

Integration tests environment setup

Testcontainers initialisation

According to the Testcontainers documentation (source: https://java.testcontainers.org/features/networking/), you cannot specify the port on which a container will be accessible on your machine. To address this limitation, we adopt a specific approach:
First, we run the containers without pre-configuring them. Then, we dynamically override local environment variables with the automatically assigned container settings.

const postgresContainer: StartedPostgreSqlContainer =  
  await new PostgreSqlContainer().start();  
  
const redisContainer: StartedRedisContainer =  
  await new RedisContainer().start();  
  
process.env.DATABASE_HOST = postgresContainer.getHost();  
process.env.DATABASE_PORT = postgresContainer.getPort().toString();  
process.env.DATABASE_USERNAME = postgresContainer.getUsername();  
process.env.DATABASE_PASSWORD = postgresContainer.getPassword();  
process.env.DATABASE_NAME = postgresContainer.getDatabase();  
  
process.env.REDIS_HOST = redisContainer.getHost();  
process.env.REDIS_PORT = redisContainer.getPort().toString();  
process.env.REDIS_PASSWORD = redisContainer.getPassword();

While this approach configures environment variables dynamically, we must also consider when specific modules and files in the Nest.js app are initialised. For instance, in our app, the DataSource is imported into AppModule through the ConfigModule and is defined in a separate file:

@Module({  
  imports: [  
    ConfigModule.forRoot({  
      isGlobal: true,  
      envFilePath: '.env',  
      load: [typeorm, typeorm_test],  
    }),  
    CacheModule.registerAsync({  
      imports: [ConfigModule],  
      useFactory: async (configService: ConfigService) => {  
        return {  
          store: redisStore,  
          database: configService.get('REDIS_DEFAULT_DB') || '1',  
          socket: {  
            host: configService.get('REDIS_HOST') || 'localhost',  
            port: parseInt(configService.get('REDIS_PORT') || '6379'),  
            password: configService.get('REDIS_PASSWORD') || '',  
            ttl: parseInt(configService.get('CACHE_DEFAULT_TTL') || '5000'),  
          },  
        };  
      },  
      isGlobal: true,  
      inject: [ConfigService],  
    }),  
    TypeOrmModule.forRootAsync({  
      inject: [ConfigService],  
      useFactory: async (configService: ConfigService) => {  
        return configService.get('NODE_ENV') === 'test'  
          ? configService.get('typeorm_test')  
          : configService.get('typeorm');  
      },  
    }),  
    CartModule,  
    ProductModule,  
    OrderModule,  
    CustomerModule,  
    HttpModule,  
  ],  
  controllers: [],  
  providers: [],  
})  
export class AppModule {  
  constructor(@Inject(CACHE_MANAGER) private cacheManager: RedisCache) {}  
  
  async onModuleDestroy() {  
    await this.cacheManager.store.client.quit();  
  }  
}

To avoid modifying production code for tests, we override the ConfigModule during TestingModule initialisation. This technique is also useful for dynamically overriding specific environment variables or even whole classes:

const moduleFixture: TestingModule = await Test.createTestingModule({  
  imports: [AppModule],  
})  
  .overrideProvider(ConfigService)  
  .useValue({  
    get: (key: string) => {  
      if (key === 'typeorm_test') {  
        return {  
          type: 'postgres',  
          host: postgresContainer.getHost(),  
          port: postgresContainer.getPort(),  
          username: postgresContainer.getUsername(),  
          password: postgresContainer.getPassword(),  
          database: postgresContainer.getDatabase(),  
          entities: ['./src/**/*.entity{.ts,.js}'],  
          migrations: ['./src/migrations/*{.ts,.js}'],  
          migrationsRun: true,  
          logging: true,  
          retryAttempts: 3,  
          keepConnectionAlive: false,  
        };  
      }  
      const value = process.env[key];  
      if (value === undefined) {  
        throw new Error(`Environment variable ${key} not found`);  
      }  
      return process.env[key];  
    },  
  })   
  .compile();

Below is presented full code of class setting up test environment. It's worth mentioning that we utilise specific .env.integration file in order do be able to manipulate the environment regardless of default .env file.
Also testcontainers require some time to set up and it might exceed the default timeout of Jest. Its decremented to increase it manually by calling jest.setTimeout(30000) or changing your Jest configuration file.

dotenv.config({ path: '.env.integration', override: true });  
jest.setTimeout(30000); // testcontainers require longer timeout for setup  
  
export class TestIntegrationSetup {  
  public static async setup(): Promise<{  
    app: INestApplication;  
    postgresContainer: StartedPostgreSqlContainer;  
    redisContainer: StartedRedisContainer;  
  }> {  
    const postgresContainer: StartedPostgreSqlContainer =  
      await new PostgreSqlContainer().start();  
  
    const redisContainer: StartedRedisContainer =  
      await new RedisContainer().start();  
  
    process.env.DATABASE_HOST = postgresContainer.getHost();  
    process.env.DATABASE_PORT = postgresContainer.getPort().toString();  
    process.env.DATABASE_USERNAME = postgresContainer.getUsername();  
    process.env.DATABASE_PASSWORD = postgresContainer.getPassword();  
    process.env.DATABASE_NAME = postgresContainer.getDatabase();  
  
    process.env.REDIS_HOST = redisContainer.getHost();  
    process.env.REDIS_PORT = redisContainer.getPort().toString();  
    process.env.REDIS_PASSWORD = redisContainer.getPassword();  
  
    const moduleFixture: TestingModule = await Test.createTestingModule({  
      imports: [AppModule],  
    })  
      .overrideProvider(ConfigService)  
      .useValue({  
        get: (key: string) => {  
          if (key === 'typeorm_test') {  
            return {  
              type: 'postgres',  
              host: postgresContainer.getHost(),  
              port: postgresContainer.getPort(),  
              username: postgresContainer.getUsername(),  
              password: postgresContainer.getPassword(),  
              database: postgresContainer.getDatabase(),  
              entities: ['./src/**/*.entity{.ts,.js}'],  
              migrations: ['./src/migrations/*{.ts,.js}'],  
              migrationsRun: true,  
              logging: true,  
              retryAttempts: 3,  
              keepConnectionAlive: false,  
            };  
          }  
          const value = process.env[key];  
          if (value === undefined) {  
            throw new Error(`Environment variable ${key} not found`);  
          }  
          return process.env[key];  
        },  
      })  
      .compile();  
  
    const app: INestApplication = moduleFixture.createNestApplication();  
  
    return { postgresContainer, redisContainer, app };  
  }  
}

Mocking External Dependencies

We also need to mock external dependencies, specifically the external API. We achieve this by using a StoreManagerPort class, which selects the appropriate adapter based on the environment. During tests, StoreManagerPort is dynamically set to MockStoreManagerAdapter:

const storeManagerProvider = {  
  provide: StoreManagerPort,  
  useClass:  
    process.env.NODE_ENV === 'test'  
      ? MockStoreManagerAdapter  
      : FakeApiStoreManagerAdapter,  
};  
  
@Module({  
  imports: [HttpModule],  
  providers: [storeManagerProvider],  
  exports: [storeManagerProvider],  
})  
export class StoreManagerModule {}

‍StoreManagerPort is an interface that ensures each adapter implements consistent methods:

export interface StoreManagerPort {  
  getCartData(cartId: string): Promise;  
  getProductData(productId: string): Promise;  
}  
  
export const StoreManagerPort = Symbol('StoreManagerPort');

In production, we use the FakeApiStoreManagerAdapter to call the external API. For tests, we use MockStoreManagerAdapter, which retrieves data from pre-configured JSON files.

This approach provides full control over data and facilitates simulating specific test cases effectively.

@Injectable()  
export class FakeApiStoreManagerAdapter implements StoreManagerPort {  
  constructor(private readonly httpService: HttpService) {}  
  
  public async getCartData(cartId: string): Promise {  
    const { data } = await firstValueFrom(  
      this.httpService.get(`https://fakestoreapi.com/carts/${cartId}`),  
    );  
    return data as Cart;  
  }  
  
  public async getProductData(productId: string): Promise {  
    const { data } = await firstValueFrom(  
      this.httpService.get(`https://fakestoreapi.com/products/${productId}`),  
    );  
    return data;  
  }  
}

@Injectable()  
export class MockStoreManagerAdapter implements StoreManagerPort {  
  private readonly logger = new Logger(MockStoreManagerAdapter.name);  
  
  public async getCartData(cartId: string): Promise {  
    const { data } = await this.getMockData(cartId, AssetType.CART);  
    return data;  
  }  
  
  public async getProductData(productId: string): Promise {  
    const { data } = await this.getMockData(productId, AssetType.PRODUCT);  
    return data;  
  }  
  
  private async getMockData(id: string, assetType: AssetType): Promise {  
    let fullPath: string;  
  
    if (assetType === AssetType.CART) {  
      fullPath = path.join(__dirname, 'cart', `${id}.json`);  
    } else {  
      fullPath = path.join(__dirname, 'product', `${id}.json`);  
    }  
  
    try {  
      const data = await fs.readFile(fullPath, 'utf-8');  
      this.logger.log(  
        `Found data: ${JSON.stringify(JSON.parse(data), null, 2)}`,  
      );  
      return { data: JSON.parse(data) };  
    } catch (e) {  
      this.logger.error(  
        `Error reading ${assetType === AssetType.CART ? 'cart' : 'product'} (id: ${id}) JSON file: ${e.message}`,  
      );  
      return { data: JSON.parse(null) };  
    }  
  }  
}

Test Suite

The code snippet below demonstrates example integration test case for the CartsController and the associated business logic. Each test case should follow a structured approach to ensure clarity and maintainability.

describe('Integration tests CartService', () => {  
  let postgresContainer: StartedPostgreSqlContainer;  
  let redisContainer: StartedRedisContainer;  
  let app: INestApplication;
  let orderRepository: Repository;  
  let customerRepository: Repository;

// SETUP THE TESTING ENVIRONMENT
  beforeAll(async () => {  
    ({ app, postgresContainer, redisContainer } =  
      await TestIntegrationSetup.setup());  
  
    await app.init();  
    
	// OBTAIN REPOSITORIES
  orderRepository = app.get(DataSource).getRepository(OrderEntity);  
	customerRepository = app.get(DataSource).getRepository(CustomerEntity);
  });  
  
  describe('Test processing carts', async () => {  
    it('should process cart correctly when customer is not present in the DB', async () => {  
      const testedCartId = '1';  
      const testedCustomerId = '1';  

	// MAKE THE API REQUEST
      const response = await request(app.getHttpServer())  
        .post('/cart/finalize')  
        .query({  
          cartId: testedCartId,  
        });  

	// CHECK IF ORDER WAS PLACED
      const orders = await orderRepository.find({  
        where: {  
          cartId: testedCartId,  
        },  
      });  

	// CHECK IF CUSTOMER WAS INSERTED
      const customer = await customerRepository.findOne({  
        where: {  
          customerId: testedCustomerId,  
        },  
      });  

	// VALIDATE ASSUMPTIONS
      expect(orders).toHaveLength(1);  
      expect(customer).toBeTruthy();  
      expect(response).toBeTruthy();  
    });  
  });  

// CLEAN UP POSTGRES DB AND REDIS CACHE AFTER EACH TEST CASE
  afterEach(async () => {  
    await app.get(DataSource).getRepository(OrderEntity).delete({});  
    await app.get(DataSource).getRepository(CustomerEntity).delete({});  
    await app.get(CACHE_MANAGER).reset();  
  });  

// CALL TERARDOWN FUNCTION TO CLEAN THE ENVIRONMENT
  afterAll(async () => {  
    await TestIntegrationTeardown.teardown(  
      app,  
      postgresContainer,  
      redisContainer,  
    );  
  });  
});

Explanation of the Test Case

Each test case is designed to follow three main steps, ensuring consistency and clarity:

1. Prepare the Test Case Environment:
- Before executing the main logic, the test case sets up any necessary initial conditions, such as inserting mock data into the database.
2. Execute the Target API Request:
- The test sends an HTTP request to the /cart/finalize endpoint with specific parameters to simulate a real scenario.
3. Validate Results:
- After the API request, the test checks the database state and compares it with the expected outcomes to validate assumptions.

Key Features and Practices

1. Clean Test Environment:
- After each test, the afterEach() block ensures a clean environment by clearing the database and resetting the Redis cache. This prevents cross-test interference.
2. Use of Mock Data:
- As the integration setup overrides external dependencies (like external APIs), there’s no need to mock API calls or data within individual test cases. This keeps the test code concise and focused. Mocked Data can be easily reused in multiple test cases.
3. Validation of Assumptions:
- The tests thoroughly verify database states (e.g., order and customer records) and API responses, including status codes and error messages, to ensure correctness.

Running tests

In order to run test you can add specific script to package.json file in your project

"scripts": {
	"test:integration": "dotenv -e .env.integration -- jest --config ./test/jest-integration.json"
}

This script will:
1. load environment variables from .env.integration file
2. run Jest based on predefined jest-integration.json config file

The jest-integration.json is looking like that:

{  
  "moduleFileExtensions": ["js", "json", "ts"],  
  "rootDir": ".",  
  "moduleNameMapper": {  
    "^src/(.*)$": "/../src/$1"  
  },  
  "testEnvironment": "node",  
  "testRegex": ".int-spec.ts$",  
  "maxWorkers": 5,  
  "transform": {  
    "^.+\\.(t|j)s$": "ts-jest"  
  },  
  "verbose": true  
}

Environment tear down

After completing the entire test suite, it is essential to tear down the test environment to release resources and ensure no lingering side effects. The tear down process involves stopping the INestApplication and the Testcontainers instances for PostgreSQL and Redis.

Here is the implementation:

export class TestIntegrationTeardown {  
  public static async teardown(  
    app: INestApplication,  
    postgresContainer: StartedPostgreSqlContainer,  
    redisContainer: StartedRedisContainer,  
  ): Promise {  
    try {  
      await app.get(DataSource).destroy();  
  
      await app.close();  
  
      await postgresContainer.stop();  
      await redisContainer.stop();  
    } catch (error) {  
      console.error('Error during teardown:', error);  
      throw error;  
    }  
  }  
}

CI/CD with Github Actions

To ensure that newly introduced features do not break existing functionality, it is essential to validate the code automatically after each pull request or push. GitHub provides a robust solution for automating this process through GitHub Actions, which allows the creation of custom workflows triggered by specific events in your repository. This enables you to run all your unit tests and integration tests seamlessly.

By configuring GitHub Actions, you can have your tests executed automatically whenever changes are pushed to specific branches, such as main. GitHub handles the setup of the application in their virtual environment, installs the necessary dependencies, and executes the tests. These environments also support Docker and Testcontainers, making them ideal for running the integration tests.

To create a CI/CD pipeline, you can add a configuration file, .github/workflows/main.yml, to your repository's root directory. The file defines the workflow for running tests and managing the pipeline.

on:  
  workflow_dispatch:  
  push:  
    branches:  
      - main  
  
jobs:  
  
  int-tests:  
    runs-on: ubuntu-latest  
    timeout-minutes: 10  
    steps:  
      - uses: actions/checkout@v4  
      - name: Set up Node.js  
        uses: actions/setup-node@v4  
        with:  
          node-version: '18'  
      - uses: pnpm/action-setup@v4  
        with:  
          version: 9  
      - name: Install dependencies  
        run: pnpm install  
      - name: Run integration tests  
        run: pnpm test:int

This configuration sets up a GitHub Actions workflow triggered manually or on every push to the main branch. The int-tests job, running on an ubuntu-latest environment, prepares the necessary tools and executes integration tests. It uses actions/checkout to get the code, actions/setup-node to set up Node.js (v18), and pnpm/action-setup for the Pnpm package manager. Dependencies are installed with pnpm install, and tests run using pnpm test:int.

The workflow stops immediately if any test or setup step fails, ensuring only valid changes are merged. GitHub's virtual environments support tools like Docker and Testcontainers, making integration testing straightforward.

By automating tests with GitHub Actions, you maintain code quality, prevent regressions, and ensure a stable, reliable application.

Conclusion

In this article, we've covered a production-tested approach for setting up integration tests in a Nest.js application. We demonstrated how to leverage Testcontainers to create a reliable and isolated test environment, mock external dependencies effectively, and utilise Jest and Supertest for robust testing. Additionally, we showed how to automate your CI/CD workflow using GitHub Actions, ensuring a stable and maintainable application deployment process.

Fully working example of the integration tests and CI/CD setup discussed in this article, are provided in repository: GitHub Repository

Feel free to explore the code, experiment with the setup, and adapt the provided practices to suit your specific project requirements. Happy testing! 🚀

‍

Author:

Stanisław Kurzyp