Senior Data Engineer

Hiring: Lead Data Engineer - Databricks

Important

Visa: USC and GC only

Location: Manhattan, New York (Onsite)

JD:

Role Overview

• Client is seeking a Lead Data Engineer with deep hands-on expertise in Databricks, SQL, Python, and
• PySpark to design, develop, and optimize scalable data engineering solutions.
• This role offers an opportunity to work on enterprise-scale data platforms supporting analytics, digital
• products, and emerging AI use cases. The candidate will drive Databricks engineering best practices,
• solve challenging technical problems, and provide technical leadership across multiple initiatives.

Key Responsibilities

• Design and develop scalable data pipelines on Databricks, following Medallion Architecture.
• Build and optimize data ingestion, transformation, and orchestration workflows for batch and
• incremental processing.
• Optimize data processing workloads, SQL queries, and compute configurations for performance
• and cost efficiency.
• Ensure data quality, observability, lineage, and reliability across production pipelines, including
• troubleshooting complex issues.
• Provide hands-on technical leadership through solution design, code reviews, reusable
• engineering patterns, and mentoring other engineers.

Required Qualifications

• 6+ years of Data Engineering experience, with substantial hands-on experience delivering
• production solutions on Databricks.
• Strong proficiency in SQL, Python, and PySpark, including distributed data processing and
• performance optimization.
• Strong understanding of Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and
• workflow orchestration.
• Familiarity with Git, CI/CD, and production deployment practices.
• Strong troubleshooting, problem-solving, and communication skills.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional certification.
• Experience working with AWS cloud services, particularly S3 and cloud-based data integrations.
• Experience with advanced ingestion patterns such as Auto Loader, Structured Streaming, or
• Change Data Capture (CDC).
• Familiarity with Databricks Asset Bundles or Apache Airflow.
• Experience with large-scale media datasets: streaming, clickstream, or advertising data.

Interested? Apply now or DM us to explore this opportunity! You can share resumes at taukir@wiseskulls.com