Senior Data Engineer

Job Description As a Data COE Sr. Data Engineer, you will have the opportunity to design and build scalable, secure, and cost-effective cloud-based data solutions. You will develop and maintain data pipelines to extract, transform, and load data into data warehouses or data lakes, ensuring data quality and validation processes to maintain data accuracy and integrity. You will ensure efficient data storage and retrieval for optimal performance, and collaborate closely with data teams, product owners, and other stakeholders to stay updated with the latest cloud technologies and best practices. Experience: Data Engineering: 7 to 10 years with high expertise in required tech stack. Notice Period: 0-3 Months allowed Role & Responsibilities: Design and Build: Develop and implement scalable, secure, and cost-effective cloud-based data solutions. Manage Data Pipelines: Develop and maintain data pipelines to extract, transform, and load data into data warehouses or data lakes. Ensure Data Quality: Implement data quality and validation processes to ensure data accuracy and integrity. Optimize Data Processing & Storage: Ensure efficient data processing of pipelines & data storage for optimal retrieval performance. Collaborate and Innovate: Work closely with data architects, data modelers, data product managers, and stay updated with the latest cloud technologies and best practices. Mentoring: Guide & groom associate data engineers & data engineers (external/internal) to ensure timely deliverables fulfilling all data engineering guidelines & standards. Technical Requirements: Programming: Python, PySpark, SQL ETL & Integration: Databricks (LDP, Lakeflow Connect, Notebooks), AecorSoft/DataSphere Data Warehousing: SCD Types, Fact & Dimensions, Star Schema, ETL Data Modeling: Dimensional modelling understanding, Erwin tool (Optional) Medallion Architecture: Bronze, Silver, Gold (Databricks Medallion Architecture) AWS Cloud Services: S3, Lambda, EC2, Glue, Redshift GenAI exposure: Genie/Kiro/Gemini etc. Scheduling Technologies: Lakeflow Jobs, Airflow Visualization (Optional): PowerBI (Optional), GenBI (Optional) Soft Skills: Problem-Solving: The ability to identify and solve complex data-related challenges. Communication: Effective communication skills to collaborate with Engineering Leads, Data Product Managers, Data Modeler, Data Analysts, and stakeholders. Analytical Thinking: The capacity to analyze data and draw meaningful insights. Attention to Detail: Meticulousness in data preparation and pipeline development. Adaptability: The ability to stay updated with emerging technologies and trends in the data engineering field.