For the past ten years, businesses and Odoo partners have increasingly recognized the capabilities of the Odoo platform.
They have discovered that Odoo can not only integrate seamlessly with existing systems but also offers improved communication through its continuously evolving XML-RPC API. The Odoo documentation provides extensive information on how to use the Odoo REST API for seamless integration.
This has led to a surge in Odoo applications designed to synchronize data with other systems such as Magento, WordPress, or even custom-built legacy solutions.
DACO: Beyond Odoo API
The Odoo REST API allows for the retrieval and manipulation of data, including fields like datetime, integer, char, and many2one. It also supports operations on multiple records and provides access to the current user's information.
There's no denying the numerous advantages of API integrations:
- Elimination of manual data processing and manipulation. For example, an invoice created in Odoo can be automatically sent to another system for payment processing, eliminating the need for manual data entry.
- Empowering business users to build automated pipelines and processes across various systems.
- Scheduled or real-time interactions between systems allow data to be fetched and updated across platforms, ensuring all systems can access and process the same data.
- Each data exchange is defined by a set of parameters determining which data is exchanged, how it is processed, and when the exchange occurs.
- 24/7 availability leads to optimized response times and eliminates data processing delays.
It is worth noting that these features and benefits also apply to Odoo Online, the cloud-based version of Odoo.
While these benefits are undeniable for small to medium-sized deployments, our experience with larger integrations for corporate clients has been less straightforward. It quickly became clear that a more robust approach was necessary.
This issue isn't specific to Odoo. It applies to all system integrations that require a high level of data accuracy, reliability, speed, and scalability when managing a significant volume of data flows and transactions.
At Portcities, we embarked on creating DACO a few years ago in response to recurring challenges presented by our corporate clients:
- Synchronizing data across multiple systems and entities.
- Processing a large number of transactions.
- Securing password management to protect sensitive data is essential for maintaining constant accessibility and optimal performance for a large number of concurrent users.
- Implementing a secure, compliant, and centralized system for data and ETL (Extract, Transform, Load) management requires unique IDs for each data point to ensure traceability and accountability.
- To ensure the security of data exchanges, each interaction must authenticate the systems involved, typically using secure API keys.
- The DACO-powered strategy also allows for better data management, including the ability to delete no longer needed data points, reducing clutter, and improving system performance.
The reality is that standard Odoo API and webhooks fall short when these requirements become essential.
Recently, a client requested a detailed presentation on the benefits of transitioning from a typical API integration approach to a DACO-powered strategy. And here are the four major key aspects that came to light:
- Data Governance and Execution Strategies
- Queuing, Sequencing, and Consuming Processes
- Performance of Data Exchanges and their Impact on System Performance
- Overall Scalability and Endurance of the Solution
This article aims to introduce these topics, equip stakeholders and process owners with the necessary tools and knowledge, and provide documentation to empower them to make informed decisions for planning and executing their integration roadmap while minimizing risks.
Benefits of DACO for Seamless Data Integration
As businesses expand their digital ecosystems, integrating Odoo with other platforms and systems becomes increasingly common. The Odoo REST API provides a way to use HTTP methods to interact with Odoo, allowing for the retrieval and manipulation of data.
While it is possible to orchestrate several exchanges between Odoo systems as well as with other third-party applications and services, it is also true that the multiplication of bi-lateral exchanges using the same communication channels comes with a thick set of downsides:
- A limited availability of each sending or receiving system.
- A capped bandwidth.
- A clear bottleneck in the processing capacity of all systems involved.
The multiplication of data exchanges using the same communication channels strains individual systems or servers, leading to availability, bandwidth, and processing capacity bottlenecks.
These limitations manifest as a series of challenges:
- Reduced System Availability: Constant data exchange demands can lead to reduced uptime for individual systems, impacting their ability to serve users and applications.
- Bandwidth Constraints: Data transfer capacity becomes a bottleneck, hindering overall performance and slowing down critical processes.
- Processing Power Strain: System resources are stretched thin, impacting responsiveness and leading to delays in data processing tasks.
As data volume and transaction rates increase, these limitations become more pronounced, resulting in:
- Data Errors and Inaccuracies: Inconsistencies and processing failures become more frequent, compromising data integrity and reliability.
- Performance Slowdowns: Data exchange speeds suffer, leading to delays, lagging systems, and a frustrating user experience.
- Overall System Instability: The cumulative impact of these issues can lead to system instability and potential outages.
To address these challenges and enable seamless data integration, businesses are turning to DACO, a centralized solution that replaces the point-to-point model with a unified platform.
DACO offers a comprehensive suite of benefits:
- Centralized Control and Management: DACO takes control, managing all data flows, routes, triggers, and events from a single, unified platform. This centralized approach provides complete visibility and control over all integration processes.
- Enhanced Visibility and Troubleshooting: DACO builds, monitors, logs, and troubleshoots all operations across connected systems. Every action is tracked, allowing for proactive error detection, rapid resolution, and improved performance.
- Orchestrating Complex Workflows: DACO empowers users to create intricate sequences of events with triggers and executions dependent on successful operations across various environments. This level of orchestration is essential for managing complex business processes that span multiple systems.
- Preserved System Resources: By offloading data synchronization tasks to DACO, businesses can free up valuable resources in Odoo and other connected systems. This allows these systems to focus on their core functionalities, such as user interaction, data management, and business logic execution.
- Smart Queuing and Processing: DACO avoids overwhelming Odoo and other systems with real-time data exchange requests. Tasks are queued and processed efficiently using external resources, ensuring smooth operations during peak loads.
DACO transforms data exchange from a complex network of point-to-point connections into a streamlined, centralized hub, empowering businesses to achieve seamless data integration, optimize operations, and enhance overall system performance and stability.
Struggling with data consistency or system uptime?
Discover how DACO can streamline your Odoo integrations.
Queuing and processing (FIFO, Broker, and Consumer)
Within multi-system synchronization, a cornerstone of a sound data strategy lies in the stringent implementation of well-defined order and sequencing.
To illustrate this principle, consider the following scenario:
A user creates a new client record in the 'res_partner' table of Odoo1. Moments later, the same user modifies the client's name and adds their address details.
In this scenario, another system necessitates the complete client information for a separate purpose.
Let's examine the potential sequence of events under a standard Odoo API integration:
CASE 1: Availability issue right after the insert
- Client data is inserted the first time on creation.
- System is busy and the sync process to the other system times out
- Client data is updated by the user.
- The update action is sent to be synced with the other system
- The other system refuses the update since the client was not created in the first place and the UPDATE process is presented for execution before the initial INSERT data creation.
Consequence: FAILURE. The new client exists inside the source Odoo1 system but does not exist inside the other system.
CASE 2: Availability issue right after the update
- Client data is inserted the first time on creation.
- Sync process sends the new data to the other system where the client is created.
- Client data is updated by the user.
- System is busy and the sync process to the other system times out.The update action is lost.
Consequence: FAILURE. The new client exists inside the source Odoo1 but is different inside the other system since the correct update was never executed there.
Let’s now look at the same scenario with DACO this time:
CASE 1: Availability issue right after the insert
- Client data is inserted the first time on creation.
- DACO queues the operation to be executed
- System is busy and the sync process to the other system times out.
- DACO keeps the insert in the queue to re-execute it later
- Client data is updated by the user.
- DACO queues the operation to be executed later in the queue after the insert that is still pending
- DACO executes the insert and the update actions in the correct order inside the other system, potentially after a few attempts.
Consequence: SUCCESS. The data inside both Odoo1 and the other system is similar.
CASE 2: Availability issue right after the update:
- Client data is inserted the first time on creation.
- DACO queues the operation to be executed
- DACO sends the data to the other system successfully.
- Client data is updated by the user.
- DACO queues the operation.
- Execution of the update fails because of system availability issues.
- DACO keeps the update order in the queue and retries until it is executed.
Consequence: SUCCESS. The data inside both Odoo1 and the other system is similar.
The previously described scenario is a simplified illustration of data synchronization, focusing on a single data endpoint.
However, transactional and ERP backends present a far more intricate landscape due to the inherent interdependencies between data elements.
For instance, transaction creation often requires the prior existence of the partner and the associated products within the system. Similarly, a delivery order cannot be generated unless the corresponding sales order has been successfully created.
These scenarios underscore the fundamental principles of FIFO (First In, First Out) processing, an absolute necessity for successful ERP data synchronization integrations.
This principle forms the core foundation of DACO integrations, ensuring a robust centralized data flow while meticulously adhering to the required parameters of order and sequence.
Performance and scalability
Within the Odoo ecosystem, API users and human users exhibit a surprising level of similarity:
- Unified Creation: Both Human users and API ones are managed through the user admin interface within Odoo.
- Authentication Alignment: Both utilize the same Odoo authentication mechanism, leveraging the filestore and encrypted cookies (or alternative SSO).
- Access Control Consistency: Both adhere to access rules defined for individual users through access rights profiles.
- Resource Utilization Parity: Both rely on the "Odoo Python Workers" resource to execute their queries.
In the end, one might as well say that the only difference is that an Odoo user is using the Front-end interface while the API 'user' is interacting with JSON instead.
However, this shared resource pool can quickly evolve into a critical concern for the following reasons:
Performance Bottlenecks: The Impact of Heavy Processing
API-level heavy processing or load spikes can significantly hinder Odoo usability for all users. Consider these scenarios:
- An Odoo worker diligently updates a distant Magento website with extensive product details.
- An API attempts to synchronize Odoo product data with Magento while a user imports many product references (SKUs) via a substantial CSV file.
- Peak website traffic puts enormous pressure on product access.
- Odoo schedulers, already known for their system impact, run concurrently with API workflows.
In each situation, Odoo's performance suffers due to overloaded resources.
Filestore Strain: The Toll of Constant Access
The Odoo filestore experiences excessive pressure from continuous access during API calls for authorization checks (GET/POST requests). The automated nature of APIs amplifies this access, potentially pushing the filestore I/O into critical territory.
Performance Penalty of Granular Access Control
While offering granularity and security, ORM access rights come at a cost. Partial data requests, intricate domains, and an abundance of SELECT statements tied to record security logic generate high volumes of back-and-forth queries between the Odoo ORM and the PostgreSQL database.
The consequences are heavy in terms of performance:
- Increased Odoo Python worker activity.
- High I/O and table locks on crucial PostgreSQL master and transaction tables.
The Challenge of Uncontrolled Third-Party Interactions
Our extensive experience with API integrations highlights a recurring issue: unreasonable third-party behavior. Overwhelming Odoo with hundreds of requests per second due to a lack of rate limiting becomes commonplace. These practices resemble either stress testing or internal DDoS attacks.
Improper Use of ORM for Data Processing
Complex API endpoints executing functions, data processing, or transformation should never utilize the Python ORM. Odoo is not designed for mass data processing, and employing it for such purposes inevitably leads to performance degradation and potential system failure.
The Inevitable Decline: Unmanageable Growth and Concurrency
As Odoo API integrations contend with combined data growth and transaction concurrency, overall system speed and efficiency plummet. The increasing number of failed transactions and system sluggishness render the solution unusable.
DACO: A Solution Designed for Performance
DACO addresses these challenges head-on. From the outset, it offers a diverse range of connectors and strategies that empower seamless integration without compromising the performance and behavior of Odoo or other connected systems. Here's how:
- Traffic-Shaped Odoo API Usage: While Odoo API remains an option, DACO implements traffic shaping to manage query volume, weight, and payload meticulously. By enforcing maximum values, we optimize resource utilization across all systems.
- Direct Database Access for High-Volume Data: Direct database access can be employed when data volume and refresh lead times become a challenge. This approach bypasses authentication mechanisms (DACO offers its own OAuth or JWT) and circumvents the ORM, Python workers, and filestore session token access.
- Dedicated Resources for Mass Data Handling: DACO facilitates mass data transfer using flat files or BigQuery systems, entirely independent of ERP computing resources for data lake or analytics purposes.
- Isolated Processing Infrastructure: DACO relies on its own containerized and pre-allocated dedicated compute resources to process data in databases or files. Odoo and the other systems are simply not impacted anymore. Anything heavy involving data processing or formatting can be entirely subcontracted to DACO where resources are dried out and allocated specifically.
By acknowledging the potential pitfalls of shared resources and leveraging its unique architecture, DACO ensures seamless integration while preserving the performance and stability of Odoo and connected systems.
Business growth and maintainability
Consolidating all ETL and data processing flows under a single system offers significant management advantages:
- Unified Core, Simplified Operations: Gone are the days of chasing scattered code across disparate systems. DACO's central core executes all integrations, eliminating the need for fragmented management.
- Declarative Logic with Data and SQL: Business logic and evolving business requirements no longer require intricate Python coding across multiple systems. DACO empowers users to define these aspects solely using data and SQL within its framework.
- Centralized Control for Seamless Updates: Initiating, upgrading, and restarting synchronizations become centralized tasks within DACO. This eliminates the time-consuming and error-prone process of attempting to simultaneously halt multiple operational systems, potentially leading to data inconsistencies.
DACO: Uninterrupted Upgrades and Maintenance
DACO facilitates data and ETL flow upgrades without disrupting production systems. This starkly contrasts with the limitations of Webhooks and APIs running within Odoo.
Upgrading anything in this scenario necessitates stopping all Odoo services and an additional shutdown for any essential Odoo upgrades.
The Inevitability of Service Interruptions with Traditional Approaches
As the number of flows, systems, and upgrade/maintenance occurrences increases, conventional Odoo API and webhook approaches become a liability.
Maintaining data coherence across systems becomes virtually impossible while ensuring service quality and uptime. The only apparent solution under this model is to halt all involved systems for upgrades and subsequent restarts, resulting in significant downtime along with data alignment issues.
DACO: The Solution for Uninterrupted Operations
DACO offers a compelling alternative. Centralizing integrations and adopting a declarative logic approach empowers users to manage data flows and business logic efficiently.
Additionally, its decoupled architecture ensures uninterrupted system operations during upgrades and maintenance, minimizing downtime and maximizing system availability.
Selecting the Right Integration Approach
Choosing between Odoo API integration and DACO requires careful consideration of several factors. The number of systems involved plays a crucial role as complexity increases with each addition.
Long-term manageability should be a key concern. Compliance and security needs are also important, with DACO's centralized control offering advantages for stricter requirements.
Data coherence and uptime are paramount for some businesses, and DACO maintains consistency while minimizing downtime during upgrades.
Finally, consider the volume and processing demands of your data. While ETL and APIs might appear straightforward initially, their impact on performance can worsen as your needs evolve and your data grows organically.
When deciding between the Odoo REST API and DACO, consider factors such as the number of systems involved, long-term manageability, compliance and security requirements, data coherence, uptime, and the volume and processing demands of your data.
DACO offers a centralized solution that addresses the limitations of traditional API integrations, ensuring seamless data integration, optimized performance, and uninterrupted operations during upgrades and maintenance.
Ready to explore a seamless integration solution for your ERP system? Contact Portcities today! Our integration experts can help you assess your needs and recommend the most effective approach for your business.