Twitter Data streaming by using pipeline in PySpark

-

Twitter data analysis using PySpark along with Pipeline

We are processing Twitter data using PySpark and we have tried to use all possible methods to understand Twitter data is being parsed in 2 stages which is sequential because of which we are using pipelines for these 3 stages Using fit function on pipeline then model is being trained then computation are being done

from pyspark import SparkContext
from pyspark.sql.session import SparkSession
from pyspark.streaming import StreamingContext
import pyspark.sql.types as tp
from pyspark.ml import Pipeline
from pyspark.ml.feature import StringIndexer, OneHotEncoderEstimator, VectorAssembler
from pyspark.ml.feature import StopWordsRemover, Word2Vec, RegexTokenizer
from pyspark.ml.classification import LogisticRegression
from pyspark.sql import Row, Column
import sys
# define the function to get the predicted sentiment on the data received
def get_prediction(tweet_text):
try:
# remove the blank tweets
tweet_text = tweet_text.filter(lambda x: len(x) > 0)
# create the dataframe with each row contains a tweet text
rowRdd = tweet_text.map(lambda w: Row(tweet=w))
wordsDataFrame = spark.createDataFrame(rowRdd)
# get the sentiments for each row
pipelineFit.transform(wordsDataFrame).select(‘tweet’,‘prediction’).show()
except :
print(‘No data’)
if __name__ == “__main__”:
if len(sys.argv) != 3:
print(“Error!! Please define host and port number”, file=sys.stderr)
sys.exit(–1)
sc = SparkContext(appName=“PySparkShell”)
spark = SparkSession(sc)
# define the schema
my_schema = tp.StructType([
tp.StructField(name= ‘id’, dataType= tp.IntegerType(), nullable= True),
tp.StructField(name= ‘label’, dataType= tp.IntegerType(), nullable= True),
tp.StructField(name= ‘tweet’, dataType= tp.StringType(), nullable= True)
])
# reading the data set
print(‘\n\nReading the dataset………………………\n‘)
my_data = spark.read.csv(‘twitter_data.csv’, schema=my_schema, header=True)
my_data.show(2)
my_data.printSchema()
print(‘\n\nDefining the pipeline stages……………..\n‘)
stage_1 = RegexTokenizer(inputCol= ‘tweet’ , outputCol= ‘tokens’, pattern= ‘\\W’)
stage_2 = StopWordsRemover(inputCol= ‘tokens’, outputCol= ‘filtered_words’)
stage_3 = Word2Vec(inputCol= ‘filtered_words’, outputCol= ‘vector’, vectorSize= 100)
model = LogisticRegression(featuresCol= ‘vector’, labelCol= ‘label’)
print(‘\n\nStages Defined…………………………..\n‘)
pipeline = Pipeline(stages= [stage_1, stage_2, stage_3, model])
print(‘\n\nFit the pipeline with the training data…….\n‘)
pipelineFit = pipeline.fit(my_data)
print(‘\n\nModel Trained….Waiting for the Data!!!!!!!!\n‘)
ssc = StreamingContext(sc, batchDuration= 3)
lines = ssc.socketTextStream(sys.argv[1], int(sys.argv[2]))
words = lines.flatMap(lambda line : line.split(‘TWEET_APP’))
words.foreachRDD(get_prediction)
ssc.start() # Start the computation
ssc.awaitTermination() # Wait for the computation to terminate

Please refer my old PySpark Posts to understand Spark Streaming concepts in details

Popular posts from this blog

Window function in PySpark with Joins example using 2 Dataframes (inner join)

-
Input details: #● File has json records #● Each record has fields: #○ user_id #○ card_num #○ merchant #○ category #○ amount #○ ts ### Below analysis to be done Sample data: +------+--------+---------+--------+----------+-------+|amount|card_num| category|merchant| ts|user_id|+------+--------+---------+--------+----------+-------+| 243| C_108| food| M_102|1579532902| U_104|| 699| C_106|cosmetics| M_103|1581759040| U_103|| 228| C_104| children| M_110|1584161986| U_103| Solution: from pyspark.sql.functions import col # this can be done without using window function creditCardData = spark.read.json(“card_transactions.json”) useridMaxSpendDF = df.groupby(‘user_id’).max(‘amount’) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“max(amount)”,”max_amount”) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“user_id”,”m_user_id”) cond = [creditCardData.user_id == useridMaxSpen...
Read more

Complex SQL: fetch the users who logged in consecutively 3 or more times (lead perfect example)

-
From the login_details table, fetch the users who logged in consecutively 3 or more times. Table Name : LOGIN_DETAILS Approach : We need to fetch users who have appeared 3 or more times consecutively in login details table. There is a window function which can be used to fetch data from the following record. Use that window function to compare the user name in current row with user name in the next row and in the row following the next row. If it matches then fetch those records. –Table Structure: drop table login_details; create table login_details( login_id int primary key, user_name varchar(50) not null, login_date date); delete from login_details; insert into login_details values (101, ‘Michael’, current_date), (102, ‘James’, current_date), (103, ‘Stewart’, current_date+1), (104, ‘Stewart’, current_date+1), (105, ‘Stewart’, current_date+1), (106, ‘Michael’, current_date+2), (107, ‘Michael’, cur...
Read more

Credit Card Data Analysis using PySpark (how to use auto broadcast join after disabling it)

-
Input details: #● File has json records #● Each record has fields: #○ user_id #○ card_num #○ merchant #○ category #○ amount #○ ts ### Below analysis to be done Sample data: +------+--------+---------+--------+----------+-------+|amount|card_num| category|merchant| ts|user_id|+------+--------+---------+--------+----------+-------+| 243| C_108| food| M_102|1579532902| U_104|| 699| C_106|cosmetics| M_103|1581759040| U_103|| 228| C_104| children| M_110|1584161986| U_103| Application: Here we will disable auto broadcast and then join 2 data frames and will see execution plan which should use broadcast join Solution: from pyspark.sql import SparkSession from pyspark.sql.functions import broadcast spark = SparkSession.builder.master(‘local[2]’)\ .appName(‘RDD_Methods_Examples’)\ .getOrCreate() print(spark.conf.get(“spark.sql.autoBroadcastJoinThreshold”)) spark.conf.set(“spark.sql.autoBroadcastJoinThreshold”, -1) pr...
Read more