Spring Integration file-input-output-channel-adaptor :Lab-5

-

We will

  • Create and test an inbound file adapter.
  • Create and test an outbound file adapter\

Here we will only be using file-inbound-channel-adapter and file-inbound-outbound-channel-adapter  channels in place of std-in-channel-adapder and stdout-channel-adapter which we have used in previous labs(Channel labs)

spring-integration-file-outbound-channel-adapter.xml

<?xml version="1.0" encoding="UTF-8"?><beans xmlns="http://www.springframework.org/schema/beans"	xmlns:int="http://www.springframework.org/schema/integration"	xmlns:int-file="http://www.springframework.org/schema/integration/file"	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 	xmlns:int-stream="http://www.springframework.org/schema/integration/stream"	xsi:schemaLocation="	http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd	http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd	http://www.springframework.org/schema/integration/stream http://www.springframework.org/schema/integration/stream/spring-integration-stream.xsd	http://www.springframework.org/schema/integration/file http://www.springframework.org/schema/integration/file/spring-integration-file.xsd">	<int-file:inbound-channel-adapter id="producer-file-adapter"       channel="messageChannel" directory="/Users/dpq/Desktop/techWorrk/fileInbound" prevent-duplicates="true">    		<int:poller fixed-rate="5000" />	</int-file:inbound-channel-adapter>		<int-file:outbound-channel-adapter channel="messageChannel" id="consumer-file-adapter" directory="/Users/dpq/Desktop/techWorrk/fileOutbound" />			<int:poller id="defaultPoller" default="true" max-messages-per-poll="5" fixed-rate="200" />	<!-- a pub/sub message channel -->	<!--<int:publish-subscribe-channel id="messageChannel" />-->	<int:channel id="messageChannel" />          </beans>

Main container loader class which will load all the beans

Startup.java

import org.springframework.context.support.ClassPathXmlApplicationContext;public class Startup {	@SuppressWarnings({ "resource", "unused" })	// @SuppressWarnings({ "resource" })	public static void main(String[] args) {		ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext(				"/META-INF/spring/spring-integration-file-inbound-outbound-channel-adapter.xml");		while (true) {		}			}}Here once we push msg file inside inbound directory "/Users/dpq/Desktop/techWorrk/fileInbound" then that message will be pushed to outbound directory "/Users/dpq/Desktop/techWorrk/fileOutbound"

Popular posts from this blog

How to change column name in Dataframe and selection of few columns in Dataframe using Pyspark with example

-
Input details: #● File has json records #● Each record has fields: #○ user_id #○ card_num #○ merchant #○ category #○ amount #○ ts ### Below analysis to be done Sample data: +------+--------+---------+--------+----------+-------+|amount|card_num| category|merchant| ts|user_id|+------+--------+---------+--------+----------+-------+| 243| C_108| food| M_102|1579532902| U_104|| 699| C_106|cosmetics| M_103|1581759040| U_103|| 228| C_104| children| M_110|1584161986| U_103| Solution: from pyspark.sql.functions import col # this can be done without using window function creditCardData = spark.read.json(“card_transactions.json”) useridMaxSpendDF = df.groupby(‘user_id’).max(‘amount’) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“max(amount)”,”max_amount”) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“user_id”,”m_user_id”) cond = [creditCardData.user_id == useridMaxSpen...
Read more

What is Garbage collection in Spark and its impact and resolution

-
Image
Garbage Collection Spark runs on the Java Virtual Machine (JVM). Because Spark can store large amounts of data in memory, it has a major reliance on Java’s memory management and garbage collection (GC). Therefore, garbage collection  (GC) can be a major issue that can affect many Spark applications. Common symptoms of excessive GC in Spark are: Slowness of application Executor heartbeat timeout GC overhead limit exceeded error Spark’s memory-centric approach and data-intensive applications make it a more common issue than other Java applications. Thankfully, it’s easy to diagnose if your Spark application is suffering from a GC problem. The Spark UI marks executors in red if they have spent too much time doing GC. Spark executors are spending a significant amount of CPU cycles performing garbage collection. This can be determined by looking at the “Executors” tab in the Spark application UI. Spark will mark an executor in  red  if the executor has spent more than 10% of the time in gar...
Read more

Credit Card Data Analysis using PySpark (how to use auto broadcast join after disabling it)

-
Input details: #● File has json records #● Each record has fields: #○ user_id #○ card_num #○ merchant #○ category #○ amount #○ ts ### Below analysis to be done Sample data: +------+--------+---------+--------+----------+-------+|amount|card_num| category|merchant| ts|user_id|+------+--------+---------+--------+----------+-------+| 243| C_108| food| M_102|1579532902| U_104|| 699| C_106|cosmetics| M_103|1581759040| U_103|| 228| C_104| children| M_110|1584161986| U_103| Application: Here we will disable auto broadcast and then join 2 data frames and will see execution plan which should use broadcast join Solution: from pyspark.sql import SparkSession from pyspark.sql.functions import broadcast spark = SparkSession.builder.master(‘local[2]’)\ .appName(‘RDD_Methods_Examples’)\ .getOrCreate() print(spark.conf.get(“spark.sql.autoBroadcastJoinThreshold”)) spark.conf.set(“spark.sql.autoBroadcastJoinThreshold”, -1) pr...
Read more