Job Description
Job Description: We are seeking a skilled and experienced Databricks Engineer with expertise in PySpark to join our team. As a Databricks Engineer, you will be responsible for designing, implementing, and maintaining highly performant data pipelines using PySpark and Azure Databricks. You will collaborate with cross-functional teams to integrate end-to-end data pipelines, ensuring the quality and consistency of data while leveraging your expertise in PySpark. Responsibilities: Designing and implementing highly performant data pipelines from multiple sources using PySpark on Apache Spark and/or Azure Databricks. Integrating end-to-end data pipelines to ensure the quality and consistency of data from source systems to target data repositories. Collaborating with project teams to support delivery of additional project components, such as API interfaces. Evaluating the performance and applicability of multiple tools against customer requirements. Working within an Agile delivery/DevOps methodology to deliver proof of concept and production implementation in iterative sprints. Integrating Databricks with other technologies, including ingestion tools and visualization tools.
Responsibilities
Knowledge, Skills, and Abilities: Proven experience working as a data engineer. Highly proficient in using the PySpark framework (Python and/or Scala). Extensive knowledge of data warehousing concepts, strategies, and methodologies. Programming experience in Scala or Python, SQL. Direct experience of building data pipelines using Pyspark (in Databricks). Hands-on experience designing and delivering solutions using Azure services, including Azure Storage, Azure SQL Data Warehouse, Azure Data Lake, Azure Cosmos DB, and Azure Stream Analytics. Experience with ingestion tools such as Azure Data Factory. Experience with Open Source non-relational/NoSQL data repositories (e.g., MongoDB, Cassandra). Databricks Certification would be a plus. Must be team-oriented with strong collaboration, prioritization, and adaptability skills.