Spring Integration

-

Introduction

I believe the Spring Integration project is a valuable tool worthy of inclusion in any Java developer’s toolbox.  Web mashups, cloud computing, RESTful and SOAP Web services, and big data are just some parts of  the modern application portfolio that often require moving, transforming, combining, separating, filtering, and otherwise slicing and dicing many types, sizes and shapes of data in and out of application environments.  In a space that often breeds home grown, complex and sloppy solutions, Spring Integration brings clarity and simplicity – all guided by well founded patterns.

EAI is the middleware that allows applications across an Enterprise to communicate and work together.  Gartner Group is credited with defining EAI as “unrestricted sharing of data and business processes among any connected application or data sources in the enterprise.”  Admittedly, EAI can sound like many things and seem like a nebulous term, but that is the reality of the situation.

EAI is anything that allows applications to work together and share data and processes.  While not always the case, the applications share data and processes that they probably weren’t originally designed to share and in ways that may not have even have been conceivable at the time of their creation.  Often, the applications run on different boxes, atop different operating systems, and distributed across the network (maybe even the Internet).  The applications may have data stored and organized in very different forms.  The applications may have been written in different programming languages.

Enterprise Integration Patterns

While it might seem that Enterprise Application Integration (EAI) defines a disparate field of middleware software, it turns out that there are some pretty common and well understood needs within this space.  For example, applications that need to communicate with each other, often speak in different data formats and there is a need to transform the data from one application format to another.  My application uses and speaks XML while your application uses and speaks JSON.  Before you can communicate with and make use of the services and data that my application provides, the EAI solution has to transform my data to JSON for your application to understand.

There are even well known solutions to many of the common needs in the EAI world.  When you have a common solution to a commonly occurring problem you have what we call a design pattern.  Within EAI, there is a relatively famous and well known collection of design patterns.  They are the Enterprise Integration Patterns as defined by Gregor Hohpe and Bobby Woolf in their book by the same name:  Enterprise Integration Patterns (Addison-Wesley, 2003).  Hohpe and Woolf are to EAI design patterns what Gamma et. al are to general design patterns.

Spring Integration (SI) is EIP Built on Spring Framework Foundation

Why are the Enterprise Integration Patterns (EIP) important to the discussion about Spring Integration framework?  Because Spring Integration is physical embodiment of the EIP  in Java and on top of the Spring Framework.  As the Spring Integration documentation states:  “Developers who have read that book should be immediately comfortable with the Spring Integration concepts and terminology.”  Spring Integration provides a lego-like collection of EIP building blocks that allow you to build your EAI solutions in a easy to think about, easy to assemble fashion.

Of course, Spring Integration is also built on top of the Spring Framework.  This allows you to leverage the dependency injected, loosely coupled, easily extended, POJO driven capabilities of Spring in your EAI solution and even incorporate other Spring technologies (AOP, security, etc.) where necessary.

we are going to look at many of the common components (and Enterprise integration patterns) you find in EAI solutions.  Specifically, we will look at Spring Integration’s

  • Message Channels
  • Adapters
  • Gateways
  • Filters
  • Transformers
  • Enrichers
  • Service Activators
  • Routers

Popular posts from this blog

Window function in PySpark with Joins example using 2 Dataframes (inner join)

-
Input details: #● File has json records #● Each record has fields: #○ user_id #○ card_num #○ merchant #○ category #○ amount #○ ts ### Below analysis to be done Sample data: +------+--------+---------+--------+----------+-------+|amount|card_num| category|merchant| ts|user_id|+------+--------+---------+--------+----------+-------+| 243| C_108| food| M_102|1579532902| U_104|| 699| C_106|cosmetics| M_103|1581759040| U_103|| 228| C_104| children| M_110|1584161986| U_103| Solution: from pyspark.sql.functions import col # this can be done without using window function creditCardData = spark.read.json(“card_transactions.json”) useridMaxSpendDF = df.groupby(‘user_id’).max(‘amount’) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“max(amount)”,”max_amount”) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“user_id”,”m_user_id”) cond = [creditCardData.user_id == useridMaxSpen...
Read more

Complex SQL: fetch the users who logged in consecutively 3 or more times (lead perfect example)

-
From the login_details table, fetch the users who logged in consecutively 3 or more times. Table Name : LOGIN_DETAILS Approach : We need to fetch users who have appeared 3 or more times consecutively in login details table. There is a window function which can be used to fetch data from the following record. Use that window function to compare the user name in current row with user name in the next row and in the row following the next row. If it matches then fetch those records. –Table Structure: drop table login_details; create table login_details( login_id int primary key, user_name varchar(50) not null, login_date date); delete from login_details; insert into login_details values (101, ‘Michael’, current_date), (102, ‘James’, current_date), (103, ‘Stewart’, current_date+1), (104, ‘Stewart’, current_date+1), (105, ‘Stewart’, current_date+1), (106, ‘Michael’, current_date+2), (107, ‘Michael’, cur...
Read more

Credit Card Data Analysis using PySpark (how to use auto broadcast join after disabling it)

-
Input details: #● File has json records #● Each record has fields: #○ user_id #○ card_num #○ merchant #○ category #○ amount #○ ts ### Below analysis to be done Sample data: +------+--------+---------+--------+----------+-------+|amount|card_num| category|merchant| ts|user_id|+------+--------+---------+--------+----------+-------+| 243| C_108| food| M_102|1579532902| U_104|| 699| C_106|cosmetics| M_103|1581759040| U_103|| 228| C_104| children| M_110|1584161986| U_103| Application: Here we will disable auto broadcast and then join 2 data frames and will see execution plan which should use broadcast join Solution: from pyspark.sql import SparkSession from pyspark.sql.functions import broadcast spark = SparkSession.builder.master(‘local[2]’)\ .appName(‘RDD_Methods_Examples’)\ .getOrCreate() print(spark.conf.get(“spark.sql.autoBroadcastJoinThreshold”)) spark.conf.set(“spark.sql.autoBroadcastJoinThreshold”, -1) pr...
Read more