Case Study: Scaling Data Insights with Secure & Efficient AI-Powered Pipelines on AWS for CogAbility

Overview: The Challenge & Cross-Border Collaboration

CogAbility, a leading U.S. provider of AI-powered digital employees known as CogBots, empowers digital workers through automation and intelligent workflows. The firm secured access to the Criminal Justice (CJIS) dataset from the U.S. government, aiming to extract actionable insights for legal research and policy analysis. However, CogAbility faced significant hurdles:

Data Sensitivity:

The CJIS data contained highly sensitive information, demanding rigorous security and compliance measures in line with U.S. government regulations, including the Criminal Justice Information Services (CJIS) Security Policy.

Data Volume & Variety:

The massive dataset encompassed thousands of diverse case files in various formats, requiring a scalable solution for efficient processing.

Regulatory Constraints:

Due to strict U.S. regulations, access to the AWS GovCloud environment (where the data resided) was restricted to U.S. citizens located within the United States.

To address these challenges, CogAbility engaged skyPurple Cloud, a UK-based cloud services team, comprised of AWS-certified AI/ML experts, Solutions Architects & Cloud Engineers.

The mission?

To design and implement a secure, scalable, and automated data pipeline that could transform raw CJIS data into structured, classified insights – all while navigating the cross-border regulatory constraints and ensuring compliance with CJIS standards.

The Solution: Secure & Efficient AI-Powered Data Pipelines on AWS

The solution combined the strengths of AWS public cloud services with US GovCloud, while prioritizing security, cost-efficiency, performance, and compliance:

1. Secure Data Ingestion (AWS GovCloud):

CogAbility's U.S.-based team securely ingested the CJIS data into an isolated AWS S3 bucket within the AWS GovCloud environment.

2. Data Processing & Classification (Commercial AWS Account):

CogAbility's U.S.-based team securely ingested the CJIS data into an isolated AWS S3 bucket within the AWS GovCloud environment.

AWS Batch:

AWS Batch was utilised for parallel processing of large batches of case files, significantly accelerating throughput and reducing processing time while optimising costs by dynamically provisioning compute resources.

Amazon Textract (Asynchronous Operations):

Textract's asynchronous operations were implemented to efficiently process the diverse document formats, enabling the pipeline to handle high volumes of text extraction requests without bottlenecks.

Amazon Comprehend (Asynchronous Operations & Custom Entities):

Asynchronous operations for Comprehend were deployed, and custom entity recognizers developed, to precisely classify documents and extract case-specific entities. Results were cached to minimize redundant API calls and reduce costs.

Amazon S3 (Intelligent-Tiering):

To further optimise storage costs, S3 Intelligent-Tiering was implemented to automatically move less frequently accessed data to lower-cost storage classes.

3. Data Transformation & Migration (AWS GovCloud)

AWS CloudFormation:

CloudFormation templates were used to automate provisioning of resources in both public and GovCloud environments. Template parameters were carefully managed to account for differences in service endpoints, ARNs, and naming conventions between the commercial AWS account and AWS GovCloud. Deploying the architecture to Public Cloud and then handing off the templates to the US-based CogAbility team for deployment to AWS US GovCloud infrastructure.

Custom Scripts:

Transformation scripts were developed to adapt the CloudFormation templates for the US GovCloud environment, ensuring a smooth and error-free deployment without having to replicate work.

4. Classified Data Output (AWS GovCloud):

The pipeline delivered structured, classified data into designated S3 buckets within the AWS US GovCloud environment, ready for further processing by CogAbility's orchestration tools like Camunda.

Compliance & Security

skyPurple Cloud, as an approved partner of Orca Security, also leveraged Orca's agentless cloud security platform to continuously monitor and ensure the pipeline's compliance with CJIS regulations, throughout the development and deployment process. Orca's comprehensive security posture management capabilities, together with “shift-left” features, provided real-time visibility into potential vulnerabilities and compliance gaps, enabling rapid remediation and ensuring the protection of sensitive CJIS data.

Results: Customer Outcomes & Satisfaction

CogAbility praised skyPurple Cloud's solution for its ability to navigate complex regulatory and technical challenges:

Security & Compliance:

The solution adhered to stringent U.S. government security protocols and data protection regulations, including CJIS.

Scalability & Efficiency:

The pipeline handled massive data volumes efficiently, ensuring timely and cost-effective processing.

Actionable Insights:

The solution transformed raw, unstructured CJIS data into structured, classified information, readily available for CogAbility's research and analysis initiatives.

Cross-Border Collaboration:

Our seamless collaboration across continents demonstrated the ability to deliver innovative solutions while adhering to strict regulatory requirements and different time zones.

Commitment to Innovation & Support:

skyPurple Cloud continues to offer CogAbility ongoing enhancements and support, highlighting our dedication to long-term customer success and continuous innovation.

Conclusion

This project exemplified skyPurple Cloud's expertise in AI/ML, AWS application architecture, and cross-border collaboration, empowering automation technology firms like CogAbility to extract valuable insights from complex datasets while adhering to rigorous security and compliance standards. Our commitment to innovation and unwavering support ensures that our clients exceed their goals and maintain a competitive edge within their respective industries.

Contact skyPurple Cloud

See how we can support your Cloud Projects.

Let's connect.

sales@skypurple.cloud

UK +44 (0) 1637 477396