Continuously investing in quality (functional programming, code quality, tests, pull request reviews, refactoring, etc.) is part of our strategy to sustainably maximize the business value we deliver.
Here's where we are now:
- We maintain a UI Library that is used in all of our frontend projects.
We sit somewhere in the middle of the “Scala as a worse Haskell” and “Scala as a better Java” spectrum. We love
functional programming and make use of libraries like
doobie, but we keep a
pragmatic stance on purity and have to play nice with the Hadoop and Spark ecosystems.
Here are some highlights of the major backend components:
http4sAPI running on Fargate that is used by any person or process interacting with our platform (e.g. the frontend, external applications, etc.).
apps: a framework for building applications on top the Narrative platform, allowing us and third parties to extend the functionality of our systems. Apps use whatever technologies makes sense for the problem at hand, but they typically look like microservices deployed to Fargate.
ingestion: Lambdas and Spark jobs that ingest supplier data and store it in our Apache Iceberg-backed "lakehouse".
transaction: Spark jobs that transact the data between the buyers and sellers.
- We operate in AWS. Our long-running services are deployed on EC2, Fargate, and Lambda, using Terraform for infrastructure automation.
- We handle a decent amount of data: our miscellaneous S3 buckets contain in excess of 2PB of data.
- Our scheduled (Spark) jobs are mostly executing on EMR and triggered by Jenkins.
- We also heavily use other AWS technologies such as DynamoDB, S3, RDS, and Step Functions.
- Other services we use include: GitHub Actions for CI/CD, DataDog for monitoring, etc.
- Being a small team, we think twice before rolling out custom infra that we need to monitor and maintain.
Workstation Setup Guide
It is also worth taking a look at our Dev Workstation Setup Guide which gives more information on the tools that we work with.