Azure Integration Platform as a Service (iPaas)

Introduction

From old days, enterprise system integration is one of the key challenge that many companies are facing, when it comes to providing better end-user experience. There are so much of enterprise integration platforms in the market to face these challenges.  Like so much else today, those platforms have been moved from on-premises data-centers into the public cloud as cloud computing providing higher-level of services that can be rapidly provisioned with minimal management effort, often over the Internet. Rather than using traditional integration technologies such as BizTalk Server, more and more organizations are using integration Platform as a Service (iPaaS) solutions. So that they don't need to buy whole product or server platform and can pay for service that they use.

The company Gartner, which does independent analysis of software tools, has identified Microsoft Azure iPaas as one of the leaders in iPaas offerings. Below figure is taken from the article that they have produced as "Magic Quadrant for Enterprise Integration Platform as a Service".

Figure 1

When would you able to use Azure Integration Services? Here are some of the most common scenarios: 

• Connecting applications inside your organization. The applications might run in your own on-premises data-center, in the cloud, or a mix of both. This kind of enterprise application integration (EAI) has been important for decades; it’s now being adapted for a hybrid world.
• Connecting applications in your organization with those of a business partner. Commonly known as business-to-business (B2B) integration, this often relies on standard formats such as Electronic Data Interchange (EDI).  Azure Integration Services supports these widely used standards.
• Connecting applications inside your organization to Software as a Service (SaaS) applications. As the business applications you buy continue to shift to SaaS, application integration requires connecting with these cloud services. For example, a business process in a hybrid environment might need to access both your on-premises SAP system and a SaaS CRM solution such as Salesforce.
• Integrating applications with Internet of Things (IoT) devices. The ever-increasing popularity of IoT raises a variety of new integration challenges. A cloud-based solution an iPaaS, is uniquely suited to address these, as it can be accessed by devices running anywhere.

source: Azure Integration Services (Referenced Article)

Microsoft Azure Integration Service as iPaaS offering

Microsoft has introduced fours components to support enterprise iPass and it lets you connect cloud and on-premises applications through unified set of cloud devices.

1. API Management, which provides an API service.
2. Logic Apps, supporting orchestration of business processes, workflows and more.
3. Service Bus, providing reliable enterprise messaging.
4. Event Grid, which allows raising and delivering events

source: azure.microsoft.com

Figure 2

API Management

With popularity of IOT devices and standard mobile devices, companies are looking to way how they can share their existing or new back end services as Rest APIs with their customers or end user. When sharing these services as APIs, certain characteristic should be considered.

• How calls to your application are authenticated
• How do you monitor and analyze how your APIs are being used
• How do you make it easy for developers to use your API
• How do you limit the number of calls your application can receive from clients
• How we can expose traditional services like SOAP API as Rest APIs.
• Restricting how many calls can come from a single source in a defined period.
• Blocking calls from specific IP addresses.
• Turning data caching on and off.
• Converting data from XML to JSON and vice-versa.
• Management features such as version and revision control and publishing APIs using admin console.

Microsoft has introduced API Management tool, which consists all of above characteristics, to manage all APIs on hassle free environment. It can expose REST and SOAP APIs from all kinds of back-end software, including applications running in the cloud and applications running on premises. The technology stack used to build that software could be either .NET, Java, or anything else. API Management works well with anything.

It also provides a Developer Portal, which can be used to learn how we can develop application on top of these Rest services and to request access to those  APIs.  It allows to mock the APIs, so that front development team can continue to do their development, even though back-end code are not yet ready.

In the modern world, pretty much everything is or probably should be exposed as an API. Doing this improves business agility and unlocks business value. But when you expose your APIs via API management, it introduces additional layer in between your customer and API. So performance should be considered when designing this. May be you can consider, deploying your Azure cloud service on the same region where your API is hosted, so that you can reduce latency.

Logic Apps

When we go back to the world of traditional enterprise integration solutions, we integrated system based on some form of triggers, such as on-premise table data insertion/deletion/modification, on FTP file upload, periodic calls to web services and so on. Developers wrote some complex custom business logic that carries out the steps in this process. In modern world, Serverless technology, what’s known as Application Platform as a Service (aPaaS) came into market as it simplifies the process of deploying code into production, scaling, capacity planning and maintenance operations may be hidden from the developer or operator. When serverless technologies are introduced, enterprise application are moved on to cloud. But still some of their components are still on-premise data centres. As a result of that, enterprise system integration were more more complex on this hybrid world.

To sort out all these challenges in Hybrid world, Microsoft has introduced Logic Apps as aPaaS service, which allows developers to configure workflows to implement each steps of business processes. Not only cloud based APIs, it can connect even to on-premise data center by configuring Azure resource for gateway. 

Logic app can access all kinds of other applications, including cloud applications, on-premises applications, and Azure Services. It provides more than 200 connectors, each providing a straightforward way to interact with a specific external service such as Twitter, Facebook, Service Bus, FTP, Salesforce, Dynamic 365, Office 365 Outlook, and many more. There are also connectors for on-premises technologies such as SharePoint, SQL Server, Oracle, and SAP. Other connectors allow access to common integration functions such as data conversions, XSLT transformations, and EDI. Whatever their requirements, developers can use these connectors to create logic apps for even complex integrations quite quickly. Below figure 3 shows you how it can access many kind of software running in many different places.

source: azure.microsoft.com

Figure 3

A logic app consists of a series of actions, each of which is a logical step in the process this workflow implements. These actions can express conditionals (if statements), loops, Periodic API calls to external connectors, and more. below figure 4 shows you an example.

Figure 4

Today lots of big companies are ready to move into cloud base solutions, but they are still waiting for that due to certain challenges that they are facing. System Integration is one of the challenge that they are facing. Microsoft has simplified this process through Logic Apps and it is hart of Microsoft Integration platform.

When you allow Logic Apps to access local on-premises applications, you should consider deploying your Azure cloud services on the same region where your local applications are running, so that you can reduce latency.

Service Bus

As a integration point from one software to another in traditional software development process , we provided Web Services and Rest based Web APIs. In order to identify changes that have been done on one system, other system depends on the polling or scheduled tasks, which call back-end APIs or Web Services. Even though there are changes or not, these service calls are triggered. This introduced so much of performance issues on the server where these services are hosted. At the other hand, if the both the system tightly coupled via Services, changes that have been done on one system are pushed to other by calling back-end services. In any case if back-end service was not available on certain time frame, then serious data synchronization issues could be introduced.

In order to handle these data synchronization and performance issues, Microsoft has introduced Azure Service Bus in modern software development world. Service Bus is a secure, reliable message broker. Asynchronous communication using this message broker architecture provides many more  advantages over making direct, synchronous calls to back-end APIs or Web Services:

• Provides load-leveling to handle bursts in workloads, using the Queue-Based Load Leveling pattern.
• Reliably tracks the progress of long-running workflows that involve multiple steps or multiple applications.
• Helps to decouple applications.
• Integrates with existing message-based systems.
• Allows work to be queued when a back-end system is not available.

Figure 5 shows you asynchronous communication between all kind of software.

source: azure.microsoft.com

Figure 5
Service Bus has two delivery modes, pull or push. In the pull model, the receiver continuously polls for new messages. Polling can be inefficient, especially if you have many queues that each receive a few messages, or if there a lot of time between messages. In the push model, Service Bus sends an event through Event Grid when there are new messages. The receiver subscribes to the event. When the event is triggered, the receiver pulls the next batch of messages from Service Bus. Therefor in order to provide proper integration point from one system to another, implementation on Service Bus should be done with Event Grid.

Event Grid

Even though Service Bus decoupled the systems and avoid unnecessary back-end API calls, resource usages and performance issues in back-end server, still client needs to do periodic calls to Service Bus to get the new messages from the queue. In order to avoid that unnecessary periodic calls, Microsoft has introduced Event Grid. Rather than requiring a receiver to poll for new messages, the receiver instead registers an event handler for the event source it’s interested in. Event Grid then invokes that event handler when the specified event occurs. Figure 6 illustrates this idea. Even Logic app can act as Receiver for these events.

source: azure.microsoft.com

Figure 6

Conclusion

All in all Microsoft has moved into list of leaders in iPaas by offering these four cloud severless technologies. Companies can manage and share their APIs with their clients using API management tool. While integrating one system with another, if any custom workflow should be applied, Logic App can be used. In order to avoid unnecessary resource usage and server calls, they can integrate system by means of Service  Bus with Event Grid. Scaling, fault tolerance, performance issues, security, capacity planning and maintenance operations are taken care by Microsoft Azure Cloud services. Together, they provide a full-featured iPaaS capable of connecting both cloud and on-premises software. As  mission-critical enterprise integration, Azure Integration Services can be considered as iPaaS for the modern hybrid world.

References

Magic Quadrant for Enterprise Integration Platform as a Service
https://www.gartner.com/doc/reprints?id=1-4X33NBH&ct=180420&st=sb
Azure Integration Services
https://azure.microsoft.com/mediahandler/files/resourcefiles/azure-integration-services/Azure-Integration-Services-Whitepaper-v1-0.pdf
Enterprise integration on Azure using message queues and events
https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/enterprise-integration/queues-events