The challenges that SCANOSS faced were quite unique, in comparison to other data retrieval businesses. As one of the first companies to provide a completely open source SCA tooling system, they have an extremely large dataset of 11 Terabytes! This means they’ve been able to index the majority of public open source code (from sources such as GitHub, Bitbucket, GitLab, RhodeCode).

In order for SCANOSS to scan the open source material and produce results in a timely manner, they needed incredibly high IOPS (Input/Output Operations per second). Such a high level of reads/writes puts incredible strain, as well as cost, on the processes involved.

SCANOSS needed the support of an AWS specialist that could provide immediate assistance. The retrieval timings and associated costs were something that had been a priority change for the team, so they knew they needed to partner with a consultancy who could have the technical understanding and innovative capabilities, as well as ask ‘fast and thorough’.

Firemind were ready to assist, with a dedicated Solutions Architect and Developer team who could not only theorise ways to reduce this IOPS burden, but also action architecture changes and costing analysis.

The focus on this project was to test storage optimised compute options in Amazon Web Services (AWS), with the ultimate goal to fully migrate to AWS. In order to achieve this, we had to prove the viability of a new architecture, one that could outperform their current production and workflows.

Due to the level of incredibly high IOPS, we knew that current AWS data services (such as Amazon RDS, Amazon Aurora and Amazon DynamoDB) would not be able to match the current performance they were experiencing, whilst providing speed and cost benefits. They were using Microsoft Azure with a dedicated disk and flat files. To combat this, we instead took the route of using Amazon S3 combined with Amazon EC2 I3 Instances.

Amazon EC2 I3 instances are the next generation of Storage Optimised instances for high transaction, low latency workloads. I3 instances offer the best price per I/O performance for workloads such as NoSQL databases, in-memory databases, data warehousing, Elasticsearch and analytics workloads.

From the utilisation of EC2 I3, we could take our sample data (40GB from the 11TB) and begin to check time and cost savings against the data updates and high IOPS. We found that we could cut down the scan and produce results from 79 minutes to around 29-31 minutes. This led to a 65% reduction in overall time against further tests, with cost savings equally reflecting the speed.

As with many of our data migration projects, we were able to quickly understand the customer’s goals and work backwards to ensure we crafted an architecture fit for purpose. Within 8 days, split across 7 weeks, we worked closely with SCANOSS and were able to deliver the high IOPS scan times and costs savings they wanted.

Both Firemind and SCANOSS worked very closely to ensure that we could quickly identify challenges in the project, as well as provide ongoing training on the benefits of EC2 I3 instances. This would ensure the SCANOSS Lead Developer would feel confident moving forwards as their Microsoft Azure environment moved to AWS.

11TB of open source data is no joke! And having a business that needs to scan, update and address edits and changes to such a vast amount takes some high transaction workloads. This project showcased the next generation usage of optimised storage solutions, with results that will have a direct effect on the future of SCANOSS and the services they provide to their customer base.