Etl Developer

Job DescriptionRoles and Responsibilities Development, customize and manage integration tools, databases, warehouses and analytical systems with the use of data related instruments/instancesCreate and run complex queries and automation scripts for operational data processing and building out Python ETL processes and writing complex SQL queries.

Test the reliability and performance of each part of a system and cooperate with the testing teamDeploying data models into production environments.

This entails providing the model with data stored in a warehouse or coming directly from sources, configuring data attributes, managing computing resources, setting up monitoring tools, etc.

Responsible for setting up tools to view data, generate reports, and create visualsMonitoring the overall performance and stability of the system.
Adjust and adapt the automated pipeline as data/models/requirements change.
Excellent understanding of ETL cycle. Analyze and organize raw data, build data systems and pipelines.

Combine raw information from different sources, explore ways to enhance data quality and reliability, interpret trends and patterns from the raw data.
Experience in using of Python/ PySpark and/or Scala for data engineering.
Understanding of data types/ handling of different data models. Good knowledge in various phases of SDLC Requirement Analysis, Design, Development and Testing on various Development and Enhancement Projects.
Desirable to have experience with Spark, Flink, Kafka, Flask, Scala, PySpark for Data engineering.

Experience with the Microsoft Azure or AWS data management tools such as Azure Datafactory, Datalake and Databricks or AWS Snowflake is a plusExperience with data visualization tools is a plus (PowerBI, Tableau).Understanding of descriptive and exploratory statistics, predictive modelling, evaluation metrics, decision trees, machine learning algorithms is a plusGood scripting and programming skills