In 2018, we worked with Oxford Retail Consultants to deliver an API and CMS to allow their clients, such as top 4 accounting firm - KPMG, to consume their consumer retail food and beverage spend data and integrate with their internal applications.

Oxford Retail Consultants approached Aduro with their vision in mind, to be able to allow their clients to access their consumer data through an API, at their own discretion rather than how they had been traditionally providing data to their clients, which was in the form of data dumps and excel spreadsheets.

Overall, the API allows clients to query spend data based on shops and demographics. One of the main challenges was ensuring fast response times using advanced caching techniques and coming up with a clear strategy and documentation on how to query, paginate and limit the data due to the sheer amount of data that could be returned in each request.

As part of the Discovery phase, Aduro assessed the feasibility of serving such a large amount of data to users and recommended a pattern that allowed users to set which type of spend data they wanted to retrieve, limiting them to a set number of records in one request via traditional pagination using limits and offsets. On top of this, caching was a key element to mitigating high response times. Aduro implemented filesystem caching which is long-lived, and only cleared manually after new spend data is uploaded. This meant that response times were drastically reduced.

Aduro recommended following the jsonapi.org methodology to ensure Oxford Retail's client had clear and concise pattern to follow to retrieve data. Furthermore, Aduro produced through documentation going over how to retrieve specific spend data via URL parameters and filter/includes as per the jsonapi.org methodology.

The original project spanned over a 4 month period, containing a lot of research and development on how to deal with such a large amount of data, bordering on what the industry refers to as "big data". Aduro and Oxford Retail have since gone onto work on further features such as a CMS to control how much data each of their clients can request and track how much data each client is accessing.