Npy8xoqt1shkmvfilwwi

ETL Framework for Data Warehouse Environments​

The Non Functional ETL Requirements

This course provides a high level approach to implement an ETL framework in any typical Data Warehouse environments. The practical approaches can be used for a new application that needs to design and implement ETL solution which is highly reusable with different data loading strategies, error/exception handling, audit balance and control handling, a bit of job scheduling and the restartability features and also to any existing ETL implementations. For existing implementations this framework needs to be embedded into the existing environment, jobs and business requirements and it might also go to a level of redesigning the whole mapping/mapplets and the workflows (ETL jobs) from scratch, which is definitely a good decision considering the benefits for the environment with high re-usability and improved design standards.

This course is a combination of standard and practical approaches of designing and implementing a complete ETL solution which details the guidelines, standards, developer/architect checklist and the benefits of the reusable code. And, this course also teaches you the Best practices and standards to be followed in implementing ETL solution.

Though this course, covers the ETL design principles and solutions based on Informatica 10x, Oracle 11g, these can be incorporated to any of the ETL tools in the market like IBM DataStage, Pentaho, Talend, Ab-intio etc.

Multiple reusable code bundles from the marketplace, checklists and the material required to get started on UNIX for basic commands and Shell Scripting will be provided.

What are the requirements?

  • Basic understanding of Data Warehouse Concepts
  • Basic understanding of ETL Concepts
  • Basic understanding of SQL commands and RDBMS concepts
  • Basic understanding of UNIX commands
  • Basic understanding of Shell/BAT Scripting

What am I going to get from this course?

  • This course provides an in detailed approach to implement an ETL framework in typical Data Warehouse environments. This approach can be used for a new application that needs to design and implement ETL solution which is highly reusable with data loading, error handling, audit handling, job scheduling and re-start-ability features. This framework will help reduce time and increase quality due to high re-usability and design standards.
  • Metadata Categories, learn the commonly used types of metadata in a real time project and how these are different from the Business and Technical viewpoints.
  • ETL Framework process flow, the process flow and different activities which should be taken care during the ETL framework implementation from file (source data) validations, Exception handling and Audit Control.
  • Data Sourcing, the different types of Data Sourcing possible in a Data Warehouse environment, different mechanisms in which the data sourcing can happen like the Scheduled events, Change Data Capture, Pub- Sub, Web services/API connectivity and the classification.
  • Different commonly required/used scripts for Data Sourcing, the different validations required to be performed for Data Sourcing and what functionality to be included in the scripts (shell/bat).
  • File Validation process, post file validation steps and file validation failure notifications.
  • Staging Layer, the need for staging layer, Reference Data, Audit columns for Staging and Reference tables, Data retention in the staging layer, partitions and DB standards.
  • Business Validation Layer, different situations possible during the data processing, concurrent workflow process, partitions in staging and business validation layer.
  • Data warehouse Layer, Dimension Load, Fact Load types/process, Fact partitions, Fact Summary Load and Source File Management/Archival.
  • Exception Handling/Error Handling, Data model for exception handling, Error Category, Error Code and different possible solutions for exception handling.
  • Sample Project Setup, Steps to download the project setup, executing the DDLs for metadata, project explanation and importing the code base into Informatica.
  • Extending the Operational Metadata’s Data Model for exception handling with additional supporting tables.
  • Error Handling Data Model, the framework for the data model design.
  • Using PMREP tables, for exception handling.
  • Audit, Balance and Control, the need, different technology components involved, table structure and data model, workflow example.
  • Configuration Management, Software Change Management, Identification, Tracking and Management of all the assets/objects of a project, One of the standard project management processes, the formal way for managing changes of the software and the process for deploying code from development to testing to production.

What is the target audience?

  • ETL Developers/Administrators
  • ETL Testing Professionals
  • Data Architects and Data Modelers
  • Data Scientists and Big Data Experts who want to understand the practical Data Warehouse Process
  • Database Administrators who want to explore the DWH/ETL/BI areas
  • BI/ETL/DW Technology experts and Team Leaders
  • Software Engineers who are already part of any Data Warehouse and Business Intelligence Projects
  • Software Engineers from different technology background who want to explore the Data Warehouse and Business Intelligence development process
  • Mainframe developers who want to switch their carrier into the Data Warehouse stream
  • Freshers/Engineering Graduates who are looking for placements
  • Non IT professionals who like to learn how data is handled in enterprises


Your Instructor


Sid
Sid

Business Intelligence Consultant and Trainer with 14+ years of extensive work experience on various client engagements. Delivered many large data warehousing projects and trained numerous professionals on business intelligence technologies. Extensively worked on all facets of data warehousing including requirement gathering, gap analysis, database design, data integration, data modeling, enterprise reporting, data analytics, data quality, data visualization, OLAP.

Has worked on broad range of business verticals and hold exceptional expertise on various ETL tools like Informatica Powercenter, SSIS, ODI and IDQ, Data Virtualization, DVO, MDM.


Class Curriculum


  ETL Framework - Process Flow
Available in days
days after you enroll
  Data Sourcing - Classification
Available in days
days after you enroll
  Script Requirements for Data Sourcing
Available in days
days after you enroll

Frequently Asked Questions


When does the course start and finish?
The course starts now and never ends! It is a completely self-paced online course - you decide when you start and when you finish.
How long do I have access to the course?
How does lifetime access sound? After enrolling, you have unlimited access to this course for as long as you like - across any and all devices you own.
What if I am unhappy with the course?
We'd hate to see you go, but if you are unsatisfied for any reason, just contact us and we'll cancel your enrollment. To avoid any misuse, the refunds are granted only to the students who have not gone beyond Section 2 of the course.
Will this help me in clear interview?
Yes all the videos can help you clear interview but you will have to relate understand the whole tool/concepts related to the course. Nothing will happen overnight and you will have to practice a lot and ask a lot of questions. We are here to help you with any questions you have at any point of time.
Which is better - videos or online classes?
Well, videos are better and elaborate. You can watch them any number of times. No limitations. And, all the videos are lifetime access with no charges to the future updates.
What is the refund policy?
If you do not like videos, full amount will be refunded provided you report within 2 days of getting the access and making the payment.
What if I miss a class during online sessions?
Please do not miss a class. In case you miss it, you can refer to videos and complete the topics.
What material will I get apart from classes?
We will support you with your questions, doubts and help you with all the presentations, documents and best practices with sample code to learn the content with more clarity.
Will it be an interactive session?
Yes, you will be able to hear trainer and communicate as well for online sessions. If you are taking the video based courses, there are options to see the questions asked by you and also by other participants. This will not only help you to get the questions clarified but also you will know the different perspectives of the subject based on the questions asked.
What all will you cover in the course?
Please look at the course content for more information.
What is the payment procedure/mode?
Click on the Enroll now button and proceed with the steps mentioned.
Do you teach on some/any real time scenarios / live projects?
Yes, all the courses and the trainings we take are real time project implementations. Everything will be practical and on sample/simulated real time project implementation.
What about trial classes for online/video based sessions?
Check our free course ‘Mini Course - Master Data Warehouse Concepts from Scratch and become an Expert’ from the home page. You can also view free session enabled for each course separately.
How is the online session conducted?
All online sessions are conducted with the help of webex/gotomeeting/teamviewer. Trainer will share his screen with you and everybody (all the participants) in the session would be able to see the screen and hear each other and talk to each other to make it an interactive session.
Will you be helping me in my project once i get into a job?
We offer, various support plans based on the need. Please write to us at info@idwbi.com or idwbitraining@gmail.com for further information on this. Please note, taking the course will not make us liable to support you for any job or job related activities.
What if you add more videos?
Once you enroll to a course (post the payments are made), all the videos are life time free and all future video updates are free of charge. We will send out an announcement regarding the new topics added on a regular basis.

Get started now!