Your Privacy

This site uses cookies to enhance your browsing experience and deliver personalized content. By continuing to use this site, you consent to our use of cookies.
COOKIE POLICY

Skip to main content

Revamping Data Pipeline Infrastructure to Increase Owner Satisfaction at Twiddy

By Patrick Kliebert

Revamping Data Pipeline Infrastructure to Increase Owner Satisfaction at Twiddy
Back to insights

In an ever-evolving technological landscape, embracing new methodologies is vital for enhancing efficiency. Our data and analytics interns recently undertook a significant overhaul of one of Twiddy’s data pipeline infrastructures, implementing Airbyte pipelines with Kestra orchestration to replace an existing Java application. Motivated by several challenges with the previous system, most importantly a complete loss of data transmission to their Owner Dashboard, this transformation has led to improved data connection management, greater reliability, and increased accessibility and efficiency.

The Old: Dependency on Java Application

Data Connections

Twiddy’s prior system depended on a Java application that managed connections to various data sources like BigQuery (BQ), PostgreSQL (PG), and Microsoft SQL Server (MS SS). These were responsible for key business areas such as the owner dashboard, guest profiles, and guest reservations.

Challenges with the Java Application

  • Complexity and Readability: One of the most daunting tasks was navigating the application’s complex and dense structure. Without adequate documentation, deciphering the functionality of each section was cumbersome. Additionally, this was the only Java code in the entire Twiddy ecosystem, creating an impediment for engineers who worked primarily with other applications.
  • Lack of Alignment with Data Processing Needs: Despite being written in Java, the application did not cater to specific data processing requirements. Inconsistencies in parsing SQL code and transmitting pipelines made it apparent that the system was not designed with data professionals in mind.
  • Stability Issues: With key components like the owner dashboard malfunctioning for months and a lack of proper documentation, the system’s interdependency led to frequent breakdowns, causing considerable challenges in maintaining the application.

The New: Implementing Airbyte Pipelines with Kestra Orchestration

Decoupling the System

Our approach for Twiddy emphasized a decoupled structure, moving away from the former Java application. If one part breaks, it doesn’t impact the rest of the system, making it easier to identify and resolve issues. Each part of the ELT (Extract, Load, Transform) process has been modularized so that a data analyst can easily create new connections to disparate destinations without any input needed from the development team.

Cloud Orchestration

Previously confined to a local server, our implementation of Airbyte pipelines orchestrated by Kestra has now shifted everything to the cloud. This move has brought unparalleled flexibility, enabling anyone in Twiddy’s organization to run the system as needed, with the added benefit that all connectors are free.

Automation

Kestra has enabled seamless integration and execution of complex tasks without manual intervention. The benefits of this automation extend beyond mere time-saving; it promotes consistency, reduces the possibility of human errors, and allows for scalable and flexible orchestration according to the specific needs of a task or system. By embracing Kestra’s automation features, Twiddy can now achieve a more efficient, reliable, and streamlined process, aligning with modern demands for agility and responsiveness in a data-driven environment.

Efficiency

By eliminating the dependencies on the old Java application, we helped Twiddy drastically reduce the time required to implement the pipelines, thus achieving a more streamlined process.

Conclusion

Twiddy’s transition from a Java-based application to Airbyte pipelines with Kestra orchestration marks an essential step towards modernity and efficiency. By addressing the previous system’s drawbacks, including challenges with readability, alignment with data needs, and stability, we have forged a more resilient and adaptable infrastructure. This improvement not only enhances Twiddy’s internal operations but also re-establishes the industry-leading owner experience they have become known for.

 

Digging In

  • Data & Analytics

    Legacy Data Modernization: A Comprehensive Guide to Upgrading Your Data Platform

    Though they may have been more than functional in the past, legacy data platforms can become a burden to your organization and prevent it from realizing its full potential. That’s why legacy data modernization can effectively transform your organization’s obsolete data systems into modern platforms that are scalable, efficient, and better equipped to handle today’s […]

  • Data & Analytics

    Masking Data 101: Safeguarding PII in Your Organization

    In today’s digital age, data security and privacy are paramount. As organizations increasingly collect, store, and process personal data, protecting Personally Identifiable Information (PII) has never been more critical. One essential practice that organizations can implement at the database level to secure this sensitive information is to obfuscate it through the usage of data masking […]

  • Data & Analytics

    Unlocking the Full Potential of a Customer 360: A Comprehensive Guide

    In today’s fast-paced digital economy, understanding your customer has never been more critical. The concept of a customer 360 view has emerged as a revolutionary approach to gaining a comprehensive understanding of consumers by integrating data from different touchpoints to offer a holistic view. A customer 360 view is about taking an overarching approach to […]

  • Data & Analytics

    Microsoft Fabric: A New Unified Data Platform

    MicroPopular data services and tools often specialize in specific aspects of the data analytics pipeline, serving teams in the data lifecycle. For instance, Snowflake addresses large-scale data warehousing challenges, while Databricks focuses on data engineering and science. Power BI and Tableau have become standard tools for business intelligence tasks. So, where does Microsoft Fabric create […]

  • Data & Analytics

    Improve Member Experience: Maximize Engagement & Value for Associations

    As you know, member engagement is key to providing value and retaining members over time. However, you must also recognize that member needs and preferences are evolving rapidly, especially as they desire more seamless digital experiences. Additionally, member expectations for personalized, omnichannel interactions have risen in recent years, and this means that associations must strategically […]

  • Data & Analytics

    A Guide to Data Strategy Success in Your Association

    While countless organizations aim to harness the potential of data, few possess a clear strategy to transform raw information into actionable insights that fuel their operations and marketing efforts. Don’t fall into the trap of investing in limited, tactical solutions.