Job Description
Senior Data Engineer Job Description Job Title: Senior Data Engineer Experience: 12+ Years
Job Type: W2 Contract
Job Summary We are seeking a highly experienced Senior Data Engineer with 12+ years of experience in designing, developing, and maintaining scalable data platforms and data pipelines. The ideal candidate will have strong hands-on expertise in Python, SQL, Spark, Databricks, cloud technologies, ETL/ELT, data warehousing, and data integration .
Responsibilities - Design, develop, and maintain scalable ETL/ELT data pipelines for enterprise applications.
- Develop complex data transformation and processing workflows using Python, SQL, and Apache Spark .
- Build and optimize data pipelines using Databricks and PySpark .
- Design and implement data solutions on AWS, Azure, or Google Cloud Platform cloud platforms.
- Develop data ingestion processes from databases, APIs, files, and other structured/unstructured sources.
- Implement data quality, validation, reconciliation, and monitoring processes.
- Design and optimize data warehouses, data lakes, and lakehouse architectures .
- Work with relational and NoSQL databases to support data engineering requirements.
- Perform performance tuning and optimization of SQL queries, Spark jobs, and data pipelines.
- Implement orchestration using Apache Airflow, Azure Data Factory, AWS Glue, or similar tools .
- Develop reusable frameworks and components for data ingestion and transformation.
- Integrate data engineering solutions with CI/CD pipelines and DevOps processes.
- Implement data security, access controls, encryption, and governance best practices.
- Troubleshoot production data pipelines and resolve data processing issues.
- Collaborate with data architects, analysts, data scientists, application developers, and business stakeholders.
- Participate in Agile/Scrum ceremonies and contribute to technical design and architecture discussions.
- Mentor junior and mid-level data engineers and provide technical guidance.
Required Skills - 12+ years of experience in Data Engineering, ETL, or related data technologies.
- Strong hands-on experience with Python and SQL .
- Extensive experience with Apache Spark / PySpark .
- Strong experience with Databricks and Delta Lake.
- Experience building enterprise-scale ETL/ELT pipelines .
- Strong knowledge of data warehousing concepts, dimensional modeling, and data lake architectures.
- Experience with one or more cloud platforms: AWS, Azure, or Google Cloud Platform .
- Experience with databases such as SQL Server, Oracle, PostgreSQL, MySQL, Snowflake, or similar .
- Experience with workflow orchestration tools such as Airflow, Azure Data Factory, AWS Glue, or similar .
- Strong understanding of batch and near-real-time data processing.
- Experience with Git, CI/CD, Jenkins, Azure DevOps, or GitHub Actions .
- Strong understanding of data quality, data validation, and performance optimization.
- Excellent analytical, troubleshooting, and communication skills.
Preferred Skills - Experience with Azure Data Lake, AWS S3, ADLS, or Google Cloud Storage .
- Experience with Snowflake or other modern cloud data warehouses.
- Knowledge of Kafka, Event Hubs, or other streaming technologies .
- Experience with Terraform or Infrastructure as Code .
- Knowledge of Docker and Kubernetes.
- Experience implementing data governance, metadata management, and lineage .
- Experience working with large-scale distributed data environments.
- Knowledge of Medallion Architecture (Bronze, Silver, Gold) .
- Experience with real-time/streaming data pipelines.
- Experience in designing highly available and fault-tolerant data platforms.
For applications and inquiries, contact:hirings@openkyber.com
Job Tags
Contract work