In order to keep your data solution running smoothly, there are a few things you should try to anticipate to give your solution the best chance of handling change driven by evolving business needs. Obviously, there is no way to predict every possible change, or prevent changes from occurring, but knowing that change is a very real possibility will help you build the best data platform for your purpose. A well-built data platform takes into account what the architect knows about the future at that particular moment in time, as well as incorporating mechanisms to support change that can’t be predicted. A generic mechanism to support changing requirements is more useful than trying to predict every possible variation, however it’s important to not let ‘perfect’ be the enemy of ‘good’!
There are three main things you should factor in when thinking about evolving company and data needs within your organisation: growth, business-as-usual change, and big changes.
Growth
A typical data warehouse grows continuously, which naturally results in more data being processed. One of the biggest causes of issues we see with longstanding data solutions is unmanaged growth. You should build your data solution with the expectation that growth will occur, and plan for this as effectively as you can. Typically, this means having an archiving strategy, factoring in things such as GDPR and internal data retention policies: it’s important to manage deletion alongside retention, to keep your data as clean as possible.
Regularly reviewing your archiving strategy (alongside your data strategy) to ensure it fits your business needs and is documented for the future is crucial to managing growth.
Another thing to consider is your underlying storage system: can it handle the level of growth you’re expecting? Will performance be impacted as more data is added? Capacity and performance are major factors for on-premises solutions, and with cloud solutions you also need to consider the cost of increased or higher performance storage.
Business-as-usual (BAU) Change
BAU Changes are the things that you know will happen, so you need to make sure your solution can cope with them. It’s inevitable that somebody will want another report or metric at some point, so there’s no point building a data platform that’s aggressively optimised for one deliverable, but will take weeks of development to change slightly. Your data platform needs to be able to cope with this level of change. Ideally, this should be thought about when first designing the platform, incorporating management strategies such as a test environment, code-based data structures, or a DevOps-based release mechanism to enable and streamline these changes. You need to be able to support BAU change to keep your data platform at the heart of your data strategy.
Big Changes
Sometimes we are affected by changes that are impossible to anticipate, and these can often be larger changes that impact the entire business. Company buy-outs or mergers, major external events (energy crisis anyone?) or even just unforeseen changes of strategic direction can hugely impact a data platforms usefulness almost overnight. Although these changes are nigh-on impossible to predict, there are steps you can take to make big changes less catastrophic. Ensuring you have documentation for the various processes and components in your data platform – and keeping these up to date and relevant – means that big changes can be assessed and reacted to much more easily. Understanding your architecture on a granular level, and using groups of modular components, managed through well documented workflows, will enable you to react to big changes in an efficient and sustainable way.
Good Management Practice
Beyond the basic housekeeping that you are, of course, carrying out for your data platform, some management best practices to minimise the impact of changing needs include:
- Monitor performance and investigate issues – fix anything that’s not just expected growth
- Carry out archiving tasks (and if automated, make sure they’ve worked!)
- Use release management to minimise impact of change
- Use change control and update documentation of what’s been changed
- Keep an eye on your costs – especially in Cloud
- Keep on top of technology – if you do have a big change, there may be a better way of implementing it that you did initially.
Ultimately, it’s important that change, however big or small, is not an afterthought for your data platform. Understanding that growth, BAU changes and big changes can affect the way you use your data in various ways can and should drive the way you use your data platform, and following some of our best practice guidance can minimise the real world impact change.
Of course, if this is all too much, our Managed Data Services are always available to look after the technical side of things, and allow you to focus on making data-driven decisions with the peace of mind that your data is in great hands.
Blog Author
Andrew Hill, Managed Services and Support Manager, Simpson Associates
Back to blog