PySpark Data Engineer

Design, develop, and maintain scalable ETL/ELT data pipelines using PySpark. Process and transform large datasets from various structured and unstructured sources. Build robust data workflows and optimize Spark jobs for performance and scalability. Develop and maintain data models, data lakes, and data warehouse solutions. Collaborate with Data Scientists, Analysts, and Business Teams to understand data requirements. Ensure data quality, integrity, security, and governance standards are maintained. Troubleshoot and resolve performance bottlenecks in Spark applications. Monitor data pipelines and automate operational processes. Implement CI/CD practices for data engineering workflows. Create and maintain technical documentation for data pipelines and processes.