One of the “headaches” multiple organizations face when implementing a cloud-based data strategy is the seemingly endless list of systems that need to be integrated into a coherent, trackable and maintainable solution. The list will vary on a case-by-case basis, but in its simplest form, a functional corporate data platform requires at least a computational engine, storage solutions, metadata engines, monitoring tools, orchestration mechanisms, and authentication/authorization services, without even mentioning regulatory compliance or governance requirements. All in all, there is no shortage of challenges to be faced when attempting such an endeavor.
Managing an extensive ecosystem of tools can be a daunting prospect, especially for firms that do not have the in-house knowledge to effectively -and securely!!- leverage all these elements in a cohesive whole that can provide real bottom-line value to its users in particular, and to their organization. It is in this critical juncture where initiatives that seemed like a good idea when envisioned are bogged down, with unforeseen costs driven by implementation challenges and integration inefficiencies that significantly reduce the real value expected from those initiatives in the first place.
There are two potential paths forward for organizations facing this juncture: use in-house teams or external partners to build a bespoke solution “perfectly” adapted to fit their exact requirements, or leverage an existing solution -or multiple, specialized ones- within a commercial-off-the-shelf strategy. Having worked on dozens of data-related projects, I’ve seen my share of both strategies implemented in a wide range of scenarios, and it has become evident that, for all their strengths and weaknesses, neither approach provides a satisfactory answer that fully exploits the potential offered by the virtually endless stack of solutions and tools offered by modern cloud providers.
Let’s start with bespoke solutions: sure, they provide an incomparable level of customization and specificity, especially if organizations are willing to pay the “premium” implied in today’s professional services tariffs. They also enable stakeholders and end-users to engage in an open discussion as to what their dream outlook would be, without having to limit themselves by constraints in the form of a rigid functional offering. In short, it is the holy grail for firms looking to build the most effective solution for their use-cases, having all their needs covered without any excessive or unnecessary surface that, in the long run, equals additional cost or complexity.
That approach, sadly, hides some troublesome compromises that progressively -and relentlessly - increase costs, hindering value delivery and operational margins for these customized products. For starters, being a one-off solution, the maintenance and upgradability responsibilities are mostly devolved into the customer organization once delivery is complete, with the alternative being a lockup with the provider for an expensive follow-up commitment. And if faced with the need to implement a new, unforeseen technical breakthrough -think AI circa 2021- it implies a completely new project engagement, with a new elicitation, planning, development, test and deployment loop that will consume important amounts of resources and, perhaps more important, time.
Now with the COTS approach: agility is the keyword for this game, since leveraging multiple prebuilt solutions -which is the usual approach, given the complexity discussed before- enables in-house developers or partners to assemble a working end-to-end platform within a shorter timeframe and with less resources than a fully customized solution. A more rigid and less dynamic requirement definition process ensues, with stakeholders and end-users constrained by the functional and non-functional limitations of the catalogue of solutions chosen, but it is still possible to deliver the basics with no frills.
Yet again, hidden weaknesses and threats also make this approach problematic. By combining multiple COTS products, integration becomes a challenge in itself, coupled with the need to manage and monitor each individual system without a cohesive framework. And as use cases become more complex, or more scopes need to be differentiated for different departments of the organization, a rational management of access and visibility across the entire spectrum of applications becomes all but prohibitively demanding for the team tasked with the management of the platform. If left unchecked, all these factors can result in an ungovernable framework that complicates -or even impedes- timely access to the data needed across the organization.
But what if I told you that there is a third way. Not as in Gordon Brown’s economic theories, but in the form of an integrated data platform that leverages the advantages of a solid catalog of services within a unified framework, but that does so without imposing unnecessary overhead in the form of unwanted services and, even more importantly, provides enough flexibility to satisfy the needs and wants from both stakeholders and end-users. This is what we have built with Sidra Data Platform, an end-to-end data platform that integrates a wide catalogue of solutions into a single, cohesive and flexible enterprise-grade solution.
Sidra provides services that handle each step of the data cycle (ingestion, validation, catalogue, etc.) using Azure’s resources, integrating what are individually strong offerings (e.g., Azure Databricks for computational workloads) into an optimized solution that derives the most value from each one of them. This architecture provides organizations with a unified framework to deploy, leverage and monitor both the services themselves and the processes they execute.
A data project built with Sidra will enable developers and engineers to tailor their solutions to the needs of the organization within a flexible functional offering, not only counting with the wide catalog of services already offered by the Platform but also having access to advanced software development toolkits (SDK) that enable them to build their own solutions within Sidra’s unified framework. With this approach, we bridge flexibility with reliability, enabling an agile approach to projects that does not sacrifice technical quality or long-term value. As for supportability, organizations that use Sidra gain access to a virtuous cycle of incremental upgrades and, more importantly, the addition of the latest technical and operational breakthroughs through our continuous-delivery cadence, where versions of our Platform are released virtually every month with new features, quality-of-life improvements and fixes that are automatically available to all our customers.
There is another advantage that provides extraordinary value for our customers by tackling the complexities of integrating multiple, sometimes completely unrelated systems: the advanced UI-based administrative solutions that Sidra offers for its customers. An organization that owns a Sidra installation has complete transparency over what services does it has, where they are installed, who can access them, and how they are being used. Answering all these questions in a traditional COTS-based scenario would require multiple system administrators just to track the inventory of applications, while a bespoke solution would need a completely customized logic and interface design -which equals tons of money- to implement something remotely like what Sidra offers out of the box.
And what about everyday processes such as setting up a new data source, or distributing a given dataset to an isolated workspace downstream? Here we have strived to provide an organic way for platform operators to configure and monitor them with UI-based tools, self-recovery protocols for potential transient failures and notifications/alerts for situations that require an action to restore nominal operation. Features such as this -as well as all those mentioned before- are the ones which allow us to confidently affirm that Sidra represents a new paradigm for a strategic approach to data-related endeavors, providing a revolutionary option for those organizations that want leverage the advantages of a unified, sustainable, agile and resilient framework for their data strategy.
If you want to hear more about us, I would be more than happy to tell you our story and what we can offer. In case a more institutional approach is needed, our sales team is ready to guide you through the possibilities Sidra offers to you and your organization.