Data Engineer - all-in-one financial data platform in NYC
What you'll do:Own the data platform. You will own the infrastructure for the data platform that powers all of the customer offerings. This includes an Apache Spark cluster, SQL Server databases, scheduling/monitoring tools, API crawlers, SFTP servers, and ad-hoc analysis services such as Apache Zeppelin and RStudio—all hosted on the cloud.
Work extensively with open source technology. You’ll work heavily within the Spark ecosystem, as well as explore new open source technologies to solve customer needs. The skills you develop here will serve you well beyond this role.
Interact with a diverse team. You will collaborate closely with other teams at to expand and improve the capabilities of the data platform. This includes working closely with the structuring team to resolve scaling or functionality pain points in data processing, the modeling team to build out a modern data science platform, and the frontend product engineering team to scale database queries to ship new products.
Gain knowledge of the financial industry. You’ll have an exclusive view into the system that enables Americans to afford houses, cars, and college. You will learn about the participants, terminology, and mathematics behind this sector of the financial markets.
What you bring:Experience with all aspects of working with big data. You have 2+ years of professional data engineering experience working with large data sets and are intimately familiar with the construction of scalable ETL pipelines, intricacies of accurate data processing, and infrastructure involved with ensuring the reliability of hundreds of daily processes.
A Spark Enthusiast. You have at least one year of professional programming experience with Apache Spark and a strong understanding of Spark internals and the operational complexities of managing a Spark cluster. You have engaged with the Spark community on mailing lists or on Github and may have contributed source code to the project.
A well-rounded engineer. You have 2+ years of professional programming experience with Scala or Java and a deep appreciation for engineering fundamentals. You understand the importance of writing tests, designing systems for long term maintainability, and evaluating both sides of common engineering considerations.
Enthusiastic about expanding your financial knowledge. You are excited to learn more about the intricacies of the financial system. You are adept at learning on the job and unafraid to dive into technical finance books.
Undergraduate or graduate degree in Finance, Math, or Engineering. Note that we’re not anti dropouts if you’re a superstar.