Python basic quick tutorial

-

I have created basic python tutorial with running examples on jupytor notebook

 

Popular posts from this blog

How to change column name in Dataframe and selection of few columns in Dataframe using Pyspark with example

-
Input details: #● File has json records #● Each record has fields: #○ user_id #○ card_num #○ merchant #○ category #○ amount #○ ts ### Below analysis to be done Sample data: +------+--------+---------+--------+----------+-------+|amount|card_num| category|merchant| ts|user_id|+------+--------+---------+--------+----------+-------+| 243| C_108| food| M_102|1579532902| U_104|| 699| C_106|cosmetics| M_103|1581759040| U_103|| 228| C_104| children| M_110|1584161986| U_103| Solution: from pyspark.sql.functions import col # this can be done without using window function creditCardData = spark.read.json(“card_transactions.json”) useridMaxSpendDF = df.groupby(‘user_id’).max(‘amount’) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“max(amount)”,”max_amount”) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“user_id”,”m_user_id”) cond = [creditCardData.user_id == useridMaxSpen...
Read more

Spring Integration Routers LAB-12

-
Image
Routers direct messages to an appropriate message channel typically based on what is in the payload or header of a message. Routers do not alter the message like a transformer. Routers don’t remove messages from the system like a filter can. They simply provide forks in the road of message channels in a Spring Integration (SI) application. In this lab, you explore a few commonly used SI routers. Specifically, in this lab you will: Configure and use an XPath router to route XML payload messages. Define a recipient router to send the same message to a collection of channels. Scenario – Route using XPath expression In previous filter Lab, we used XPath expressions in filters to remove or discard some XML messages from the system. In a similar fashion, as will be demonstrated in this part of the lab, an XPath router uses an XPath expression to determine which message channel receives a message containing XML in its payload. In particular, note that the application already contains an inbo...
Read more

What is Catalyst query optimizer?

-
Image
Spark SQL was designed with an optimizer called Catalyst based on the functional programming of Scala. Its two main purposes are: first, to add new optimization techniques to solve some problems with “big data” and second, to allow developers to expand and customize the functions of the optimizer. Catalyst Spark SQL architecture and Catalyst optimizer integration Catalyst components Los componentes principales del optimizador de Catalyst son los siguientes: The main components of the Catalyst optimizer are as follows: Trees The main data type in Catalyst is the tree. Each tree is composed of nodes, and each node has a nodetype and zero or more children. These objects are immutable and can be manipulated with functional language. As an example, let me show you the use of the following nodes: Merge ( Attribute ( x ), Merge ( Literal ( 1 ), Literal ( 2 )) Where: Literal(value: Int) : a constant value Attribute(name: String) : an attribute as input row Merge(left: TreeNode, right: TreeNo...
Read more