Unlock the Power of Informatica(IICS) with Parallel Taskflows

Table of Contents

1. Introduction
2. What is a Data Flow Task?
3. Objects used in the Data Flow Task
   • Source Tables
   • Target Tables
   • Mappings
4. Creating Mapping Tasks
   • Mapping Task for Data Flow 1
   • Mapping Task for Data Flow 2
5. Task Flow
   • Overview of Task Flow
   • Creating a Task Flow
   • Adding Mapping Tasks to Task Flow
   • Validating and Saving the Task Flow
6. Running the Task Flow
   • Monitoring Subtasks in Jobs
7. Parallel Task Execution
8. Linear Task vs Sequential Task
9. File Watcher Mals
10. Conclusion

Data Flow Task: An Overview

The Data Flow Task is a critical component in the workflow of data integration processes. It allows the movement and transformation of data from source tables to target tables through the use of mappings. In this article, we will explore the concepts and functionalities of the Data Flow Task, including the objects involved, the creation of mapping tasks, task flow management, and the execution of parallel tasks. We will also discuss the difference between linear and sequential tasks and touch upon the use of File Watcher Mals. So, let's dive in and understand the intricacies of the Data Flow Task in Detail.

1. Introduction

The Data Flow Task plays a crucial role in the ETL (Extract, Transform, Load) process. It enables the seamless transfer of data between different tables, databases, or systems by applying various transformations along the way. Whether it is filtering, sorting, aggregating, or joining data, the Data Flow Task facilitates complex data operations efficiently.

2. What is a Data Flow Task?

A Data Flow Task is a component in data integration tools that allows the movement and transformation of data from source tables to target tables. It consists of multiple mappings that define the flow of data within the task. Each mapping specifies the source, target, and transformations to be applied to the data.

3. Objects used in the Data Flow Task

Before diving into the functionalities of the Data Flow Task, let's familiarize ourselves with the objects used within it:

Source Tables

Source tables are the tables from which data is extracted. These tables can be located in a database or any other source system. In our example, we have two source tables: "test one" and "test two."

Target Tables

Target tables are the tables where the transformed data is loaded. These tables can be located in the same or different database as the source tables. In our example, we have two target tables: "test three" and "test four."

Mappings

Mappings define the flow of data from source to target tables. They specify the transformations to be applied and the rules for data manipulation. In our example, we have two mappings: "M data flow one" and "M data flow two." The first mapping loads data from "test one" to "test three," and the Second mapping loads data from "test two" to "test four."

4. Creating Mapping Tasks

In order to execute the mappings within the Data Flow Task, we need to Create mapping tasks. These mapping tasks act as containers for the individual mappings and define their runtime environments. Let's create the mapping tasks for our example:

Mapping Task for Data Flow 1

The first mapping task, "MTM mapping task_data flow one," runs the "M data flow one" mapping. It is responsible for executing the data flow from "test one" to "test three" within its runtime environment.

Mapping Task for Data Flow 2

The second mapping task, "MTM mapping task_data flow two," runs the "M data flow two" mapping. It is responsible for executing the data flow from "test two" to "test four" within its runtime environment.

5. Task Flow

The task flow within the Data Flow Task determines the sequence of execution of the mapping tasks and controls the overall workflow. Let's understand how to create and manage a task flow.

Overview of Task Flow

A task flow is a collection of mapping tasks that can be executed sequentially or in parallel. In our example, we will create a parallel task flow that executes both mapping tasks simultaneously.

Creating a Task Flow

To create a task flow, navigate to the "New" option and select the "Task Flow." Give a Meaningful name to your task flow; for example, "TF_MTM data flow."

Adding Mapping Tasks to Task Flow

Within the task flow, add the desired mapping tasks. In our example, we will add "MTM mapping task_data flow one" and "MTM mapping task_data flow two" to execute both mappings.

Validating and Saving the Task Flow

After adding the mapping tasks, validate the task flow to ensure its correctness. Once validated, save the task flow for future use.

6. Running the Task Flow

To execute the task flow, we need to run it within the ETL tool. Let's see how to monitor the execution of subtasks in the jobs.

Monitoring Subtasks in Jobs

Navigate to the jobs section and find your task flow job. Monitor the subtasks within the job to ensure they are running in parallel. Once all subtasks are completed successfully, the task flow will receive a success status.

7. Parallel Task Execution

The Data Flow Task enables parallel execution of multiple mapping tasks, resulting in faster processing of large volumes of data. By executing mappings in parallel, time and resources are optimized, and data integration pipelines become more scalable.

8. Linear Task vs Sequential Task

While parallel task execution allows the simultaneous execution of multiple mapping tasks, linear and sequential tasks follow a different approach. In the next video, we will explore the differences between linear and sequential tasks and understand when to use each of them in the data integration process.

9. File Watcher Mals

Apart from parallel and sequential task execution, the ETL tool also offers the functionality of File Watcher Mals. File Watcher Mals can be used to trigger a task flow when specific files are detected or undergo changes in a particular directory. This feature enhances automation and real-time data processing capabilities.

10. Conclusion

The Data Flow Task is a vital component in data integration processes. It allows for the seamless movement and transformation of data between source and target tables. By understanding the objects involved, creating mapping tasks, managing task flows, and executing parallel tasks, You can enhance the efficiency and scalability of your data integration pipelines.