Tech Studio Online

Transformation: With transformation, we get a new RDD. There are many ways to achieve this, such as: Input in a Hadoop file system (such as HDFS, Hive and HBase) to create a RDD. Convert the parent RDD to get a new RDD. Create a distributed RDD using the data on a single machine through parallelize or makeRDD functions (Difference: A) The makeRDD function provides the location information of the data, which is not provided in the parallelize function. B) The returned values of both functions are ParallelCollectionRDD. But in parallelize function, you can specify the number of partitions, while in the makeRDD function, it is fixed as the size of the seq parameter). Create a RDD based on the DB (MySQL), NoSQL (HBase), S3(SC3) and data streams. Action: With action, we get a value or a result (directly caching the RDD to the memory) All the transformations adopt the lazy strategy. The computation won’t be triggered if you submit a transformation function only. Instead, the computation...

Create a recipient list router: Recipient list routers are meant to disburse received messages to all of its outbound channels without regard for message header or payload content. If you will, a recipient list router simply blasts incoming messages to all the channels listed as “recipients.” In this step, you create a recipient router to route Norway ship orders to both a file adapter and a service activator that prints out information regarding the Norway ship order to the Console view. Add two new Norway message channels. There needs to be two new message channels to serve as recipients of the Norway messages. The first new channel will take messages that will ultimately be routed to the file system. The second new channel will receive messages that will ultimately be routed to the service activator. <int:channel id=”norwayFileChannel” /> <int:channel id=”norwaySAChannel” /> Change the File adapter for the Norway messages to now get its messages fr...

Routers direct messages to an appropriate message channel typically based on what is in the payload or header of a message. Routers do not alter the message like a transformer. Routers don’t remove messages from the system like a filter can. They simply provide forks in the road of message channels in a Spring Integration (SI) application. In this lab, you explore a few commonly used SI routers. Specifically, in this lab you will: Configure and use an XPath router to route XML payload messages. Define a recipient router to send the same message to a collection of channels. Scenario – Route using XPath expression In previous filter Lab, we used XPath expressions in filters to remove or discard some XML messages from the system. In a similar fashion, as will be demonstrated in this part of the lab, an XPath router uses an XPath expression to determine which message channel receives a message containing XML in its payload. In particular, note that the application already contains an inbo...

Channels are an integral part of any Spring Integration application. There are many channels to choose from. Understanding the basic channel types (subscribable and pollable) is essential to any Spring Integration developer. In this lab, you explore the basic channel types and see the default channel at work. Specifically, in this lab you will: Create a new Spring Integration Maven Project Define the Spring configuration file. Define a Java SE class for running/testing your Spring Integration components. Create and test a subscribable channel. Create and test a pollable channel. Create and test the default Spring Integration channel – the direct channel

Adapters are endpoints that connect a message channel to an external system or technology. Adapters take content from outside of SI and bring it in as messages into SI channels or take SI messages and deposit their content to an external system. Specifically, in this lab you will: Create and test an inbound file adap Create and test an outbound file adapter.

Filters allow, on the basis of a message’s content or metadata (in the message header), a message to pass from one channel to the next or reject and discard the message from the system – that is, not allowing the rejected message into the next channel.  Not all messages are of interest. That is, there is often a need to filter out some messages that enter a channel. Applications may only want to get messages formatted in a certain way (XML or JSON). Applications may only be interested in certain types of data (new sales messages but not messages about shipments). Other application may be interested in messages with certain message headers (for example when a message was created – only wanting to look at messages created after 5pm). In this lab, you explore Spring Integration (SI) filters to examine messages in a channel and accept those of interest and discard the others. Specifically, in this lab you will: Implement the SI MessageSelector interface and configure an SI filter....

XML is popular way to deliver data. However, Java applications prefer to work with the data in object form. A built-in SI transformer provides the ability to convert an XML payload message into a message containing a Java object holding the data of the XML message. This process is called unmarshalling transformation (an opposite process – going from Java object to XML payload also exists and that is called marshalling). Under the covers, the unmarshalling transformer uses JAXB technology to perform the XML to object work. In this portion of the lab, you explore the use of a SI unmarshalling transformer. We will use JAXB here to marshalling and un-marshalling.

As with filters and many other SI components, you can create your own custom transformer. When the transformation is particularly complex or when you need to transform to / from a type that Spring does not know about, you will find custom transformation the route to take. In this next step, you create a custom transformer – one that transforms the string payload of the source message to a pig Latin translation of the string in the target message. In this step, you also see the use of annotations to configure your transformer. SI (and all of Spring) allows the use of annotations in your Java code to simplify the configuration of components and reduce the amount of XML associated to your application. While annotations were not used in the prior labs (like the filter lab), you will find that SI comes with a number of annotations that can be used in place of XML for the configuration of just about any SI component. Examine and annotate the Transformer class. A class containing the Pig Lati...

No, Spring does not have Autobots, but the concept – turn something from one thing to another thing – is provided for in Spring Integration. Spring Integration (SI) transformers turn one type of message into another. In the world of integration, data providers and data consumers don’t always speak the same language. So transformers provide SI applications the means to convert messages between formats to facilitate non-homogeneous message exchange. For example, a producer may provide information in XML format. It is the data your application needs, but it would like that data in JSON form. A SI transformer can perform that message conversion. In this lab, you explore several types of SI transformers – some provided by SI out of the box. As you have learned with other SI components, you can also create your own custom transformer. Specifically, in this lab you will: Configure and use a message payload transformer. Define a custom transformer. Explore the use of annotations to reduce...

Scenario – Transform String Messages In the first part of this lab, you some simple string message to string message transformers. That is, these transformers take a message with string payload from a channel (called the source message in SI), change the string payload of a message, and put a message with the altered string into another channel (called the target message in SI). These simple examples will help you understand the basic configuration of a transformer and allow you to see how to create your own custom transformer.