Tech Studio Online

Here we will try to understand Spring Integration(SI) using examples and will understand and cover each and every component in detail.

Spring Integration is an open source framework for enterprise application integration. It is a lightweight framework that builds upon the core Spring framework. It is designed to enable the development of integration solutions typical of event-driven architectures and messaging-centric architectures.

We will Create and test an inbound file adapter. Create and test an outbound file adapter Before we start we already explore adapters. we have already used adapters in the first lab. You used Standard Input and Standard Output adapters to bring String data into the message channel and print it to the Console view Here we will only be using file-outbound-channel-adapter channel in place of stdout-channel-adapter which we have used in previous labs(Channel labs) spring-integration-file-outbound-channel-adapter.xml <?xml version="1.0" encoding="UTF-8"?><beans xmlns="http://www.springframework.org/schema/beans" xmlns:int="http://www.springframework.org/schema/integration" xmlns:int-file="http://www.springframework.org/schema/integration/file" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int-stream="http://www.springframework.org/schema/integration/stream" xsi:schemaLocation=" http://www.spr

The only difference b/w Pollable and Direct channel is , There will not be any queue size limit in Direct channel <int:channel id=”messageChannel” /> Everything else will remain same as with pollable channel. spring-integration-channels.xml <?xml version="1.0" encoding="UTF-8"?><beans xmlns="http://www.springframework.org/schema/beans" xmlns:int="http://www.springframework.org/schema/integration" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int-stream="http://www.springframework.org/schema/integration/stream" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd http://www.springframework.org/schema/integration/stream http://www.springframework.org/schema/integration/stream/spring-integration-stream.x

Subscribable channels don’t buffer messages and deliver the messages to any and all subscribers. Pollable channels, can buffer messages and deliver the message to a single subscriber. If there are more than one subscribers, it picks the first subscriber and skips the others. In this step, you replace the subscriber channel with a pollable channel to see the effect on the SI application. We will have same example of lab1 with minor changes. those changes are Remove Subscribable channel and add Pollable channel Remove channel: <int:publish-subscribe-channel id=”messageChannel” /> Add Channel: <int:channel id=”messageChannel” > <int:channel id="messageChannel" > <int:queue capacity="2"/></int:channel> So all xmls would be like below: spring-integration-channels.xml <?xml version="1.0" encoding="UTF-8"?><beans xmlns="http://www.springframework.org/schema/beans" xmlns:int="http://www.springframew

Here we will understand publisher and subscriber interaction b/w channels using java project with scenarios. Pom.xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.intertech.si</groupId> <artifactId>lab1-channels</artifactId> <version>0.0.1-SNAPSHOT</version> <name>Spring Integration Lab 1</name> <description>Lab 1 - using channels</description> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <springframework.version>4.0.4.RELEASE</springframework.version> <spring.integration.version>4.0.0.RELEASE</spring.integration.version> <spring.integration.xml.version>4.0.0.RELEASE</spring.integration.x

Spring Integration enrichers are really just a special type of transformer (see previous post in this series). Enrichers take a message from a channel and enhance it by adding information to its header or payload. Thus, an enricher is a transformer that only adds information to the message rather than alter it in other ways.

In this installment of our Spring Integration tutorial, we examine routers. Routers distribute messages to one or more message channels. Some routers (content routers) examine the message payload or headers in order to select a particular destination message channel. Other routers (recipient list routers) simply distribute the message to all listed message channels. Example Content Router: <int:payload-type-router input-channel="routingChannel"> <int:mapping type="java.lang.String" channel="stringChannel"/> <int:mapping type="java.lang.Integer" channel="integerChannel" /></int:payload-type-router> Recipient List Router: <int:recipient-list-router id="customRouter" input-channel="routingChannel"> <int:recipient channel="channel1" selector-expression ="payload.equals('foo')"/> <int:recipient channel="channel2" selector-expression="

Transformers take a message from a channel and creates a new message containing converted payload or message structure.  This allows the provider of information and the consumer of information to communicate via SI without having to settle on a common format.  XML can be transformed to JSON, JSON transformed to Java objects, etc. Transformers convert payload or structure of a message into a modified message Above image converting XML message to Json. Spring Integration Transformers: Transformers comes up with in-build transformers XML to Object(Unmarshallers) or Object to XML(Marshallers) Object to String / String to Object file to string / string to file Object Serialiser or deserialiser Object to Map / Map to Object Object to Json /Json to Object Claim check It uses simple POJOs to create your own custom transformer Simple Transformers examples: Object to String transformer: Takes message with an object payload from inbound channel calls toString method of a payload object put messag

Filters allow, on the basis of a message’s content or metadata (in the message header), a message to pass from one channel to the next or reject and discard the message from the system – that is, not allowing the rejected message into the next channel. The filter is a “yea or nay” component determining which messages flow through and which messages do not. Filters are again endpoints that sit between channels that allows or rejects messages from one message channel to other. Filters allows some messages to pass from one channels to another channel. Messages not selected are discarded Selection occur based on message payload or message metadata(header information) As with Adapters, SI provides many filters out of the box We can create out own filter with its own custom message selection criteria SI Built-In Filters Spring Integration provides several ready to use filters with the framework , you merely have to configure them to use Built-In filters include Expression filter , Xpath filt

Adapters are messaging endpoints.  In particular, they are the message endpoint that bridges your Spring Integration system to external systems and services (like JMS queues, databases, mail systems, etc.). Adapters (aka Channel adapters) An endpoint which connects channel to external systems It provides the bridge b/w Intregation framwork and external systems and services provides saperation of messaging concerns from transports and protocol used Adapters are Inbound and Outbound Inbound- Those that brings messages into Spring Integration Channels Outbound- Those that gets messages from SI channels to outside Applications , Databases etc. Built in Adapters Stream Adapters(like standard input and output stream adapters) File Adapters JMS Adapters JDBC and JPA Adapters FTP and Secure FTP( SFTP) Adapters Feed (RSS,Atom etc) Adapters Mail Adapters MongoDB Adapters UDP Adapters Twitter Adapters We can create custom adapter as per application requirements. Example Adapters: A JMS Inbound Ad

The main components Essentially in Spring Integration System or for that matter any Enterprise Application integration system, There are 2 major components , those are Producers also known as Senders and Consumers also known as Receivers and these 2 high level components communicate with each other they do so using pipes also called as channels. In Spring Integration Prouducers and consumers are known as Endpoints and pipes are known as Channels and communication occurs via Messages through Channels . Message contains Data information or some sort of event details. To understand more on this, lets discuss another use case You have an application that provides customer data and perhaps customer data provided in XML format, and there is some sort of consuming application which want this customer data but this application like to have customers who have contacted in last year(filtered contacts) and also consumer want this data in json format, here we can achieve it using filter in payl

Introduction I believe the Spring Integration project is a valuable tool worthy of inclusion in any Java developer’s toolbox.  Web mashups, cloud computing, RESTful and SOAP Web services, and big data are just some parts of  the modern application portfolio that often require moving, transforming, combining, separating, filtering, and otherwise slicing and dicing many types, sizes and shapes of data in and out of application environments.  In a space that often breeds home grown, complex and sloppy solutions, Spring Integration brings clarity and simplicity – all guided by well founded patterns. EAI is the middleware that allows applications across an Enterprise to communicate and work together.  Gartner Group is credited with defining EAI as “unrestricted sharing of data and business processes among any connected application or data sources in the enterprise.”  Admittedly, EAI can sound like many things and seem like a nebulous term, but that is the reality of the situation. EAI is an

Describe the following code and what the output will be. case class User(userId: Long, userName: String)case class UserActivity(userId: Long, activityTypeId: Int, timestampEpochSec: Long)val LoginActivityTypeId = 0val LogoutActivityTypeId = 1private def readUserData(sparkSession: SparkSession): RDD[User] = { sparkSession.sparkContext.parallelize( Array( User(1, "Doe, John"), User(2, "Doe, Jane"), User(3, "X, Mr.")) )}private def readUserActivityData(sparkSession: SparkSession): RDD[UserActivity] = { sparkSession.sparkContext.parallelize( Array( UserActivity(1, LoginActivityTypeId, 1514764800L), UserActivity(2, LoginActivityTypeId, 1514808000L), UserActivity(1, LogoutActivityTypeId, 1514829600L), UserActivity(1, LoginActivityTypeId, 1514894400L)) )}def calculate(sparkSession: SparkSession): Unit = { val userRdd: RDD[(Long, User)] = readUserData(sparkSession).map(e => (e.userId, e)) val userActivityRdd: RDD[(Long, UserAct

Question 1: What are ‘partitions’? A  partition  is  a super-small part of a bigger chunk of data . Partitions are based on  logic  – they are used in Spark to manage data so that the minimum network encumbrance would be achieved. You could also add that the process of  partitioning  is used to derive the before-mentioned small pieces of data from larger chunks, thus optimizing the network to run at the highest speed possible. Question 2: What is Spark Streaming used for? You should come to your interview prepared to receive a few Spark interview questions since it is quite a popular feature of Spark itself. Spark Streaming is responsible for scalable and uninterruptable data streaming processes. It is an extension of the main Spark program and is commonly used by Big Data developers and programmers alike. Question 3: Is it normal to run all of your processes on a localized node? No, it is not. This is one of the most common mistakes that Spark developers make – especially when they’re

1) If 8TB is the available disk space per node (10 disks with 1 TB, 2 disk for operating system etc. were excluded.). Assuming initial data size is 600 TB. How will you estimate the number of data nodes (n)? Estimating the hardware requirement is always challenging in Hadoop environment because we never know when data storage demand can increase for a business. We must understand following factors in detail to come to a conclusion for the current scenario of adding right numbers to the cluster: The actual size of data to store – 600 TB At what pace the data will increase in the future (per day/week/month/quarter/year) – Data trending analysis or business requirement justification (prediction) We are in Hadoop world, so replication factor plays an important role – default 3x replicas Hardware machine overhead (OS, logs etc.) – 2 disks were considered Intermediate mapper and reducer data output on hard disk – 1x Space utilization between 60 % to 70 % – Finally, as a perfect designer we n

1. Explain the correlation between Hadoop and Big Data? Whether you are a fresher or an experienced candidate, this is one Big Data interview question that is inevitably asked at the interviews. You need to explain that Hadoop is an open-source framework that is used for processing, storing, and analysing complex unstructured data sets for deriving actionable insights. 2. Define the terms HDFS and YARN along with their respective components. This is another Hadoop related question that you might face at your next Big Data interview. Explain that HDFS is Hadoop’s default storage unit which is mainly responsible for storing different types of data in a distributed environment. There are two components of HDFS: •  Name Node  – Contains the metadata information for all the data blocks in the HDFS. •  Data Node  – Mainly acts as substitute node and is responsible for storing the data. YARN in the context of Big Data refers to Yet Another Resource Negotiator. It is primarily responsible for

In this article, we will discuss some top 10 interview questions in Spring Boot. These questions are a bit tricky and trending heavily, nowadays, in the job market. 1) What does the @SpringBootApplication annotation do internally? As per the Spring Boot doc, the @SpringBootApplication annotation is equivalent to using @Configuration, @EnableAutoConfiguration, and @ComponentScan with their default attributes. Spring Boot enables the developer to use a single annotation instead of using multiple. But, as we know, Spring provided loosely coupled features that we can use for each individual annotation as per our project needs. 2) How to exclude any package without using the  basePackages  filter? There are different ways you can filter any package. But Spring Boot provides a trickier option for achieving this without touching the component scan. You can use the  exclude  attribute while using the annotation   @SpringBootApplication . See the following code snippet:1 @SpringBootApplication(