Data Engineer

Job Description: As a Data COE Data Engineer, you will have the opportunity to design and build scalable, secure, and cost-effective cloud-based data solutions. You will develop and maintain data pipelines to extract, transform, and load data into data warehouses or data lakes, ensuring data quality and validation processes to maintain data accuracy and integrity. You will ensure efficient data storage and retrieval for optimal performance, and collaborate closely with data teams, product owners, and other stakeholders to stay updated with the latest cloud technologies and best practices. Experience: Data Engineering: 3 to 7 years with good expertise in required tech stack. Notice Period: 0-3 Months allowed Technical Requirements: Programming: Python, PySpark, SQL ETL & Integration: Databricks (LDP, Lakeflow Connect, Notebooks), AecorSoft/DataSphere Data Warehousing: SCD Types, Fact & Dimensions, Star Schema, ETL Data Modeling: Dimensional modelling understanding, Erwin tool (Optional) Medallion Architecture: Bronze, Silver, Gold (Databricks Medallion Architecture) AWS Cloud Services: S3, Lambda, EC2, Glue, Redshift GenAI exposure: Genie/Kiro/Gemini etc. Scheduling Technologies: Lakeflow Jobs, Airflow Visualization (Optional): PowerBI (Optional), GenBI (Optional) Role & Responsibilities: Design and Build: Develop and implement scalable, secure, and cost-effective cloud-based data solutions. Manage Data Pipelines: Develop and maintain data pipelines to extract, transform, and load data into data warehouses or data lakes. Ensure Data Quality: Implement data quality and validation processes to ensure data accuracy and integrity. Optimize Data Processing & Storage: Ensure efficient data processing of pipelines & data storage for optimal retrieval performance. Collaboration: Be an integral part of the delivery team by exhibiting close coordination within the assigned team to ensure timely completion of deliverables fulfilling all data engineering guidelines & standards. Soft Skills: Problem-Solving: The ability to identify and solve complex data-related challenges. Communication: Effective communication skills to collaborate with Engineering Leads, Data Product Managers, Data Modeler, Data Analysts, and stakeholders. Analytical Thinking: The capacity to analyze data and draw meaningful insights. Attention to Detail: Meticulousness in data preparation and pipeline development. Adaptability: The ability to stay updated with emerging technologies and trends in the data engineering field.