Usage Landscape Enterprise Open Source Data Integration

Talend

Description

Enterprise data integration needs are growing exponentially over
time, as is the interest in open source technologies and the adoption
of open source solutions.
With this in mind Talend conducted a survey to define the usage
landscape of open source data integration and to profile users of
this technology. The data used in this analysis was collected from
1013 survey participants. Responses came primarily from the U.S.
(56.5%), followed by Europe (35.2%), with the rest of the responses
(8.3%) originating in the rest of the World.

Sujets

• ETL
• Open source
• Informatique

Informations

• Publié par: lettier
• Publié le: 03 octobre 2011
• Nombre de lectures: 194
• Langue: English

Extrait

WHITE PAPER

 

Usage Landscape Enterprise Open Source Data Integration  

 

 

rt. ......docuitno................................................ ............

Table of Contents   In 3   Background .................................................................... 3   Diverse Data Integration Proje..c.t..s. .................................... 4   Data Integration Needs and T.o..o.l.s. .................................... 6   Open Source Data Integration vs. Proprietary Solutions .........8.   ..... Enterprise Requirements .................................................... 9   Community Support ........................................................ 10   Community Involvem.e..n.t. .............................................. 11   Conclusion ................................................................... 13    

 

 

 

 

Talend White Paper Usage Landscape - Enterprise Open Source Data Integration    

Introduction

Background

 

Enterprise data integration nes edare growing exponentially over  time, as is the interest in openu srcoe technologies and the adoption  of open source solutions.   With this in mind Talend conducted a survey to define the usage landscape of open source data integration and to profile users of this technology. The data used in this analysis was collected from 1013 survey participants. Responses came primarily from the U.S. (56.5%), followed by Europe (35.2%), with the rest of the responses (8.3%) originating in the rest of the World.  

 

8%

35% 57%

US Europe Other     Survey respondents’ demographics

As companies merge, acquire new applications, and build their IT platforms by incorporating disparate applications with legacy systems, information systems are becoming more and more heterogeneous. As a result, tda aintegration tools are now  indispensable if enterprise IT departments are to properly manage the flows of data across the information system.   Page 3 of 13

   

Talend White Paper Usage Landscape - Enterprise Open Source Data Integration    

In addition, alternative models of software deployment—such as Software as a Service (SaaS)—and the need for interoperability with partners, customers, providers, etc., all have an important impact on data integration requirements.   The global economy is imposing cost controls on IT Managers, both in Data Integration The process of combining data residing t   erms of staff and software, at a time when data integration at different sources and providing the user with a unified view of these data. represents an increasingly larger percentage of the enterprise IT budget. Asked to do more with less, IT personnel would be better off spending cycles on tasks orththean the time consuming manual scripting needed to meet custom requirements. In fact, software resources with lower acquisition and operation costs would allow IT Managers to more easily deploy enterprise-grade solutions.   In this context, open source lustoions offer a very compelling argument. Open source tools can automate and maintain tasks formerly requiring manual scriptasn, d the existing skills of the IT  implementation team easily transfer to an open source offering. In addition, IT departments don’t have to justify significant up-front fees.

Diverse Data Integration Projects Data integration is the collectiver tme for technologies that include  ETL (Extract-Transform-Load) for business intelligence and data warehousing, and operation data integration—the flows of data across operational applications and systems. These needs can range from high throughput batch transfers of data to near-real-time, trickle-feed data flows.   Project Type Consistent with the global data integration market distribution— whether open source or proprietarmyo—st of the survey participants  (61.5%) use open source solutions for their ETL projects, in   Page 4 of 13

 

Talend White Paper Usage Landscape - Enterprise Open Source Data Integration    

particular for BI, Data warehousing and analytics. This can be attributed to the fact that ETL   tishe most mature segment of the entire data integration market.  

ETL Data Loading Operational Data Integration: Batch Migration Operational Data Integration: Real Time Database Synchronization

0% 10% 20% 30% 40% 50% 60% 70%     Types of projects for which open source data integration is used     Data Loading Data loading (41.9%) and data migration (26.5%) are the second and T a h p e p   lic p a r ti o o c n e   ss o   r  of d   a l t o a a b d a i s n e g —   fo d r a   ta e   xa in m   p a le n     fourth most popular type of project. Both of these are good iopr r to its deployment. candidates for open source solutsi, oans they are typically one-offs,  T D h a e t   a M pr i o g c r e a s t s i   on of transferring data with no ongoing purpose that would justify a long-term investment o b t e h t e w r  ee s n y   st d e a m t s a , b   a w s i e t s h ,     t a h p e  pli p c u a r ti p o o n s s e     o o r f     in an expensive proprietary tool. replacing a system with another.   Data synchronization (19.1%) is also a popular type of project Data Synchronization T c h o e n   sist p e ro n c c e y  ss  o o n f    e r s e ta m b o l t is e  hing s   ou d rc a e ta s    conducted by open sourcdae ta integration users.   continually harmonizing the data over   time. Batch vs. Real-Time Operational data integration—whether batch or real-time—is also a good fit for open source solutions. As business tempos speed up, real-time and nearly real-time operational data integration projects will prevail over bulk transfer projects. As of the date of the survey, 40% of participants used open source tools to manage their batch operational data integration tasks, compared to only 22.9% for real- time projects—but the latter is a much faster growing segment.   Page 5 of 13

 

Talend White Paper Usage Landscape - Enterprise Open Source Data Integration    

  ETL vs. Operational Data Integration Taken together, batch and real-time operational data integration projects (62.9%) are slightly better represented than ETL usage share (61.5%), even though the former market segment is less mature. And, if we also add in data synchronization, the operational project share reaches 82%. The reason for this over-representation is simply that open source tools are particularly appropriate for operational projects because they meet a number of data integration requirements, whereas—traditionally—proprietary tools focus on ETL. In addition, enterprises that want to diversify their data integration tools are often doisucraged by the licensing costs of  proprietary applications. Open usrcoe solutions offer a greater breadth of connectivity and more flexibility in terms of adoption, deployment, and maintenance.

Data Integration Needs and Tools Although software companies e artrying to provide unified integration solution packages, tdhaeta integration needs for most enterprises are so complex thateythoften need to multiply the number and nature of the integration software products they use.  

 

Manual scripting

Database utilities

Commercial software

0% 10% 20% 30% 40% 50% 60%     Data integration technologies usine dc onjunction with open source   

  Page 6 of 13

Talend White Paper Usage Landscape - Enterprise Open Source Data Integration    

 

Survey participants proved to use a combination of commercial applications, open sourcseolutions, and database utilities to meet their data integration needs.   The statistics show that using open source and commercial solutions in combination is very common (31.2%), and that the two can, and do, coexist on the same platform. In fact, open source solutions are often complementary to an existipnrgoprietary solution that—for whatever reason—cannot address a specific need. In some cases it may be that it’s not worth the expense of investing in a proprietary solution extension.   The high incidence of database utilities shown in the survey results (53.9%) is as expected—these utilities are a no-cost solution and are usually included with the databases. Their usefulness, however, is limited to dedicated database usage.   Applications are often stacked as needs arise—increasing connectivity issues—whether enterprises want their CRM system to communicate with their ERP module, or to have their disparate databases exchanging information with their home-grown platform.   Faced with multiple connectivity issues, enterprises often have no option other than manual scripting to keep data flowing across their heterogeneous enterprise systems. This is why the survey results rank manual scripting as one of the technologies most frequently invoked (54.7%) by enterprises mtoeet their integration needs. Although this is much higher than commercial (31.2%) packaged technologies, it is not surprising that manual scripting is the solution of choice as it carries the lowest initial cost.   Although manual scripting is oftiennt ended to be a short-term fix  for interchange issues, once in production it often becomes a permanent solution. And, in the end, this simple stop-gap can   Page 7 of 13

Talend White Paper Usage Landscape - Enterprise Open Source Data Integration    

become an entire home-grown platform. The drawback of hand coding or home-grown platforms surfaces over time in the inevitable maintenance problems that increase the TCO. The advantage, however, is that it fits a particular need that none of the available commercial or open source solutions can meet.

Open Source Data Integration vs. Proprietary Solutions In an ongoing effort to lower trh ediata integration software TCO,  many enterprises are now considering open source solutions, not just for one-time projects, but a lfsor their ongoing mission-critical processes, to replace or complement their expensive CPU- dependent solutions.  

 

Ease of use Performance Avoid lock-in No licensing costs Source code access

0% 20% 40% 60% 80% 100% Very important Important Neutral Not important     Decision criteria   Open source solutions are a real alternative to the proprietary world. Key players have made major strides toward improving the usability and friendliness of open source technologies, traditionally a weak spot for these applications.   In just a few short years, open source has evolved from something “geeky” into an enterprise-readsyolution. Today, open source solutions are sufficiently feateu-rich to meet complex user requirements. The survey results reflect these expectations.   Page 8 of 13

Talend White Paper Usage Landscape - Enterprise Open Source Data Integration    

Respondents felt most strongly about ease-of-use (59%) and performance (53.9%) as the most important aspects of an open source data integration solution.   Surprisingly, licensing cost is ntohte  gating criterion for enterprises  turning to open source solutio nIts .actually comes fourth after  performance, ease of use, and no lock-in (42.5%), with only 42.1% of respondents considering it very important.   Access to the source code comes last on most priority lists when enterprises are choosing open source tools.   It is a common misconception thcaotntrol of the source code is important for users of open source software. Most users today understand that open source solutions are as mature as their proprietary counterparts and, thefeorre, don’t feel the need to  enhance the code themselves.   Today, open source solutions are advantageously replacing the source code escrow of proprietary software. However, few enterprises want to allocate in-house resources (or even have the expertise) to edit, enhance, anmdaintain their data integration applications code.

Enterprise Requirements An analysis of the survey data indicates that users expect the same performance and enterprise-scale features from open source solutions that they previously fod uonnly in proprieatry products. In  order of importance tehse features include:  •   centralized scheduling and execution dashboard •   shared repository •   administration tools

 

  Page 9 of 13

Talend White Paper Usage Landscape - Enterprise Open Source Data Integration    

70% 60% 50% 40% 30% 20% Scheduling tool 10% Dashboard Shared repository 0% Administration  tool     Enterprise open source data integration requirements   First, 60.5% of respondents want a scheduling tool that lets them consolidate and centralize theirc thenical processes. Second, 57.8%  users need a dashboard to centrally monitor processes as they execute. Because enterprise users often work in teams and need to share data on large-scale projects, 54.9% consider a shared repository essential. Finally, 38.4% of enterprise users want an administration tool to centrally manage users and projects.   However, not all companies have enterprise-scale requirements. Single users and SMBs might not need that sort of enterprise-grade feature. What emerges is that o pseonurce solutions address diverse  needs for a variety of user profiles, whether large or small.

Community Support As shown, enterprises want the same support with open source solutions that commercial aipcpaltions provide. The major  difference lies in the fact that a significant number of open source users (84.9%) would rather call on the community for help addressing issues than get support from a dedicated service. This lets them reduce the cost of support and decrease their data integration budget; the return they get from the community is   Page 10 of 13

 

Talend White Paper Usage Landscape - Enterprise Open Source Data Integration    

comparable in quality to traditional support from a proprietary vendor.  

Community support (forums, etc.) Email-based or Web-based support Guaranteed response times Phone support

0% 20% 40% 60% 80% 100%     Community vs. commercial support expectations   Open source users value the forum and the other community tools at their disposal, as well as the ease-of-mind that comes from knowing that there is no pressure to upgrade or to buy new tools. The community also tends to be morse proensive than traditional support  services and community tools are no-cost to the enterprise.   However, enterprise users working on mission critical projects, do need (and demand) vendor-provided, enterprise-grade technical support. This still represents a mritnyo of the total number of users  of open source data integration (20., 9b%u)t is a fast growing  proportion.

Community Involvement Two-thirds of the respondents sayatththey are willing to actively participate in the community, and nearly half are ready to help beta-test open source productOs.p en source communities have a  real, live QA lab of thousands at their disposal. Open source users appreciate getting support from the community and feel at ease in sharing their experiences and helping other users solve problems. Getting involved in the communeitnysures the sustainability of the

 

  Page 11 of 13