Project Background

A Quantitative Research & Development firm approached us for consultative support in designing a bespoke, scalable solution to manage its growing portfolio of quantitative research products and services. The firm offered sophisticated financial models, financial data analytics, and investment insights to a wide range of clients, including mid-sized financial institutions and individual UHNW investors.

Their existing workflow was unable to cope with the computational overhead of real-time data processing from multiple market data feeds and alternative data sources, consumed manually at the time into spreadsheet software. Additionally, the firm required a custom CRM and ERP solution to support their financial portfolio management and analytics offerings, this decision being primarily motivated by cost at scale, the nature of their workflows being too bespoke and expensive to implement with existing off-the-shelf CRM and ERP options available to them in the market.

The primary objectives of the client were to:

1. Integrate real-time market and alternative data feeds, enabling the firm to enhance its quantitative research capabilities and provide richer insights to its clients.
2. Create a unified system that could manage financial portfolio research and client interactions via a custom-built CRM and ERP system.
3. Ensure scalability for future growth as the firm expanded its client base, product offerings, and employee head count.
4. Create a multi-tenant solution featuring scoped, tenant-level data security and isolation.

We began by analyzing the firm’s current infrastructure. Their legacy systems struggled with integrating third-party data feeds and supporting real-time analytics for portfolio management. Additionally, the current CRM and ERP systems were largely off-the-shelf solutions that were expensive to customize and scale due to additional licensing and SI costs, making it considerably difficult for the firm to manage customer relationships and internal operations to the standard they required.

Key findings included:

1. System Fragmentation: The firm’s quantitative research and development tooling, CRM, and ERP were isolated from each other, creating data silos and process inefficiencies.
2. Lack of Scalability: The current systems could not handle the growing volume of data or the increasing complexity of the firm’s research products.
3. Security Gaps: The system needed to be more secure, particularly with respect to handling sensitive financial data and managing multiple clients under a multi-tenant architecture.

Based on this analysis, and the firm's explicit request to build the solution on Amazon Web Services (AWS), we recommended building an architecture involving the following AWS services:

1. Amazon RDS for managing relational data, allowing for structured storage of client portfolios, CRM records, and ERP transactions.
2. AWS Lambda for serverless data processing on a pay-per-use basis, without the cost and management burden of dedicated compute infrastructure. AWS Lambda Functions would also be used to develop consumers for market data feeds and alternative data sources that the firm relied on for quantitative analysis.
3. Amazon S3 for secure and scalable storage of datasets and research outputs.
4. AWS Cognito to manage user authentication, and together with Lambda Authorizers, ensure secure, scoped tenant-level access to data.

The architecture we recommended was designed with flexibility, scalability, and security in mind, with a focus on unifying the firm’s operations and allowing it to expand its product offerings with minimal disruption.

1. Data Storage: We proposed Amazon RDS to store structured data related to the firm’s research products, client portfolios, CRM, and ERP dependencies.
2. Business Logic and Data Processing: AWS Lambda was recommended to handle serverless execution of the firm’s data processing workflows, including analytics, portfolio management updates, and other critical computations.
3. AWS ECS Fargate: Some of the firm's workloads were resource-intensive and lasted from several hours to multiple days, with varying compute, memory, and ephemeral storage requirements. Given that the firm’s developers were already familiar with Docker, Amazon ECS (Elastic Container Service) was a natural choice to orchestrate containerized workloads. AWS Fargate, as the serverless compute engine for containers, provided flexibility by allowing the firm to run containers without managing the underlying infrastructure. However, for long-running or resource-heavy tasks, careful configuration of Fargate’s task size (compute and memory allocation) was recommended to avoid resource limitations and ensure cost efficiency.
4. Multi-Tenant Support: AWS Cognito was recommended to manage user authentication and ensure secure, segregated access to the platform. Each client (institutional or individual) would have their own data partition in RDS, allowing them to interact with their portfolio and research tools securely, while IAM policies would restrict access to sensitive internal operations.
5. Security and Compliance: The firm required a solution that adhered to strict security and compliance guidelines, particularly due to its dealings with financial institutions. Encryption at rest and in transit was to be implemented using AWS Key Management Service (KMS) to manage encryption keys securely. Additionally, AWS CloudTrail and AWS Config were incorporated into the architecture to provide comprehensive auditing, real-time monitoring, and resource compliance checks, ensuring adherence to internal policies and regulatory requirements.

Though the solution was still in the design phase, we provided clear recommendations for how to deploy it effectively:

1. Infrastructure as Code: We advised using Terraform or AWS Cloud Development Kit (CDK) to define and provision the infrastructure declaratively, enabling predictable and consistent deployments across different environments.
2. CI/CD Pipeline: To support continuous integration and deployment, we recommended the implementation of AWS CodePipeline and CodeDeploy. This would allow for fast, reliable updates to the platform without downtime, ensuring that new features and bug fixes could be rolled out in a controlled manner.
3. Security Best Practices: We recommended implementing a multi-layered security approach to safeguard the firm’s infrastructure. This included fine-grained IAM policies and roles to enforce the principle of least privilege, as well as security groups and Network Access Control Lists (NACLs) to tightly regulate inbound and outbound traffic across the network. A well-architected VPC design was proposed to isolate critical workloads, ensuring that only trusted sources had access to sensitive resources. To ensure ongoing protection, we advocated for regular penetration testing and security audits to proactively identify and mitigate potential vulnerabilities, ensuring continuous alignment with industry security standards and regulatory requirements.
4. Testing & Validation: We recommended a testing strategy that included performance tests to validate the scalability of real-time data processing, security tests to ensure that sensitive client data remained secure, and user acceptance tests to ensure that the CRM and ERP systems met the firm’s operational needs.

Through this engagement, we provided the client with a clear roadmap for modernizing their technology infrastructure. By proposing a cloud-native solution built on AWS, we addressed key operational challenges, including system fragmentation, scalability limitations, and security considerations.

The recommendations positioned the firm to:

1. Unify Operations: By integrating portfolio management, CRM, and ERP functions into a single platform, the firm would be able to operate more efficiently and provide a higher level of service to its clients.
2. Scale Effectively: The proposed architecture would allow the firm to handle increasing data volumes and client demands without sacrificing performance.
3. Enhance Research Capabilities: Native support for data integrations via AWS Lambda consumers would enable the firm to deliver richer and more timely insights to its clients.
4. Improve Security: By adhering to AWS’s Well-Architected and Security Reference frameworks, the firm would ensure compliance with financial regulations while protecting sensitive client data.