Spring Integration Labs

-

Here we will try to understand Spring Integration(SI) using examples and will understand and cover each and every component in detail.

Popular posts from this blog

How to change column name in Dataframe and selection of few columns in Dataframe using Pyspark with example

-
Input details: #● File has json records #● Each record has fields: #○ user_id #○ card_num #○ merchant #○ category #○ amount #○ ts ### Below analysis to be done Sample data: +------+--------+---------+--------+----------+-------+|amount|card_num| category|merchant| ts|user_id|+------+--------+---------+--------+----------+-------+| 243| C_108| food| M_102|1579532902| U_104|| 699| C_106|cosmetics| M_103|1581759040| U_103|| 228| C_104| children| M_110|1584161986| U_103| Solution: from pyspark.sql.functions import col # this can be done without using window function creditCardData = spark.read.json(“card_transactions.json”) useridMaxSpendDF = df.groupby(‘user_id’).max(‘amount’) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“max(amount)”,”max_amount”) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“user_id”,”m_user_id”) cond = [creditCardData.user_id == useridMaxSpen...
Read more

What is Garbage collection in Spark and its impact and resolution

-
Image
Garbage Collection Spark runs on the Java Virtual Machine (JVM). Because Spark can store large amounts of data in memory, it has a major reliance on Java’s memory management and garbage collection (GC). Therefore, garbage collection  (GC) can be a major issue that can affect many Spark applications. Common symptoms of excessive GC in Spark are: Slowness of application Executor heartbeat timeout GC overhead limit exceeded error Spark’s memory-centric approach and data-intensive applications make it a more common issue than other Java applications. Thankfully, it’s easy to diagnose if your Spark application is suffering from a GC problem. The Spark UI marks executors in red if they have spent too much time doing GC. Spark executors are spending a significant amount of CPU cycles performing garbage collection. This can be determined by looking at the “Executors” tab in the Spark application UI. Spark will mark an executor in  red  if the executor has spent more than 10% of the time in gar...
Read more

Spring Integration Routers LAB-12

-
Image
Routers direct messages to an appropriate message channel typically based on what is in the payload or header of a message. Routers do not alter the message like a transformer. Routers don’t remove messages from the system like a filter can. They simply provide forks in the road of message channels in a Spring Integration (SI) application. In this lab, you explore a few commonly used SI routers. Specifically, in this lab you will: Configure and use an XPath router to route XML payload messages. Define a recipient router to send the same message to a collection of channels. Scenario – Route using XPath expression In previous filter Lab, we used XPath expressions in filters to remove or discard some XML messages from the system. In a similar fashion, as will be demonstrated in this part of the lab, an XPath router uses an XPath expression to determine which message channel receives a message containing XML in its payload. In particular, note that the application already contains an inbo...
Read more