Overview: The Challenge & Cross-Border Collaboration
CogAbility, a leading U.S. provider of AI-powered digital employees known as CogBots, empowers digital workers through automation and intelligent workflows. The firm secured access to the Criminal Justice (CJIS) dataset from the U.S. government, aiming to extract actionable insights for legal research and policy analysis.
However, CogAbility faced significant hurdles that required innovative solutions and expert cloud architecture.
The Challenges
Data Sensitivity
The CJIS data contained highly sensitive information, demanding rigorous security and compliance measures in line with U.S. government regulations, including the Criminal Justice Information Services (CJIS) Security Policy.
Data Volume & Variety
The massive dataset encompassed thousands of diverse case files in various formats, requiring a scalable solution for efficient processing.
Regulatory Constraints
Due to strict U.S. regulations, access to the AWS GovCloud environment (where the data resided) was restricted to U.S. citizens located within the United States.
The Mission
To design and implement a secure, scalable, and automated data pipeline that could transform raw CJIS data into structured, classified insights—all while navigating the cross-border regulatory constraints and ensuring compliance with CJIS standards.
To address these challenges, CogAbility engaged skyPurple Cloud, a UK-based cloud services team comprised of AWS-certified AI/ML experts, Solutions Architects & Cloud Engineers.
The Solution: Secure & Efficient AI-Powered Data Pipelines on AWS
The solution combined the strengths of AWS public cloud services with US GovCloud, while prioritizing security, cost-efficiency, performance, and compliance:
1. Secure Data Ingestion (AWS GovCloud)
CogAbility's U.S.-based team securely ingested the CJIS data into an isolated AWS S3 bucket within the AWS GovCloud environment, ensuring compliance with all regulatory requirements from the outset.
2. Data Processing & Classification (Commercial AWS Account)
- AWS Batch: Utilized for parallel processing of large batches of case files, significantly accelerating throughput and reducing processing time while optimising costs by dynamically provisioning compute resources.
- Amazon Textract (Asynchronous Operations): Textract's asynchronous operations were implemented to efficiently process the diverse document formats, enabling the pipeline to handle high volumes of text extraction requests without bottlenecks.
- Amazon Comprehend (Asynchronous Operations & Custom Entities): Asynchronous operations for Comprehend were deployed, and custom entity recognizers developed, to precisely classify documents and extract case-specific entities. Results were cached to minimize redundant API calls and reduce costs.
- Amazon S3 (Intelligent-Tiering): To further optimise storage costs, S3 Intelligent-Tiering was implemented to automatically move less frequently accessed data to lower-cost storage classes.
3. Data Transformation & Migration (AWS GovCloud)
- AWS CloudFormation: CloudFormation templates were used to automate provisioning of resources in both public and GovCloud environments. Template parameters were carefully managed to account for differences in service endpoints, ARNs, and naming conventions between the commercial AWS account and AWS GovCloud.
- Custom Scripts: Transformation scripts were developed to adapt the CloudFormation templates for the US GovCloud environment, ensuring a smooth and error-free deployment without having to replicate work.
The architecture was deployed to Public Cloud and then the templates were handed off to the US-based CogAbility team for deployment to AWS US GovCloud infrastructure.
4. Classified Data Output (AWS GovCloud)
The pipeline delivered structured, classified data into designated S3 buckets within the AWS US GovCloud environment, ready for further processing by CogAbility's orchestration tools like Camunda.
Compliance & Security
skyPurple Cloud, as an approved partner of Orca Security, also used Orca's agentless cloud security platform to continuously monitor and ensure the pipeline's compliance with CJIS regulations, throughout the development and deployment process.
Orca's comprehensive security posture management capabilities, together with "shift-left" features, provided real-time visibility into potential vulnerabilities and compliance gaps, enabling rapid remediation and ensuring the protection of sensitive CJIS data.
Results: Customer Outcomes & Satisfaction
CogAbility praised skyPurple Cloud's solution for its ability to navigate complex regulatory and technical challenges:
Conclusion
This project exemplified skyPurple Cloud's expertise in AI/ML, AWS application architecture, and cross-border collaboration, empowering automation technology firms like CogAbility to extract valuable insights from complex datasets while adhering to rigorous security and compliance standards. Our commitment to innovation and unwavering support ensures that our clients exceed their goals and maintain a competitive edge within their respective industries.