Senior Data Engineer - CreditNinja
We are looking for a Senior Data Engineer to join a team of analytics and machine learning experts. The hire will be responsible for building tooling to support analytics, helping to extend our machine learning platform, expanding and optimizing our data pipeline architecture, supervising junior engineers, and interfacing with the Development team to create cross-team solutions. The ideal candidate is an experienced data engineer and data wrangler who enjoys optimizing data systems and building them from the ground up. They must be self-directed and comfortable supporting the data needs of multiple teams, systems, and products. Experience in analytics and statistics is a major bonus. The right candidate will be excited by the prospect of optimizing or even re-designing, our company’s data architecture to support our next generation of products and data initiatives; as well as mentoring and guiding junior members of the team.
Responsibilities for Sr. Data Engineer
· Supervise junior members of the data engineering team. Guiding, planning, and reviewing the team’s work.
· Create and maintain optimal data pipeline architecture.
· Assemble large, complex data sets that meet functional / non-functional business requirements.
· Extend our machine learning platform by designing tools that interface with cloud services, our current code base, and provide new flexibility in model building.
· Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, Python, and AWS.
· Build analytics tools to provide actionable insights into key business performance metrics, as well as supporting the needs of the analytics team.
· Create data-handling tools for analytics and data scientist team members that assist them in building and optimizing our decision-making process.
Qualifications for Sr. Data Engineer
· Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL), as well as working familiarity with a variety of databases.
· Experience building data pipelines, architectures, and data sets from raw, loosely structured data.
· A history of focusing on test driven design and results for repeatable and maintainable processes and tools.
· Experience building processes supporting data transformation, data structures, metadata, dependency, and workload management.
· Working knowledge of message queuing, stream processing, and highly scalable data stores.
· Strong project management and organizational skills and the ability to work independently in a fast-paced, quickly changing environment. Ability to keep up with several projects at once and understand the impact of projects within a larger system.
· Experience supporting and working with cross-functional teams in a dynamic environment.
· Experience managing junior engineers and guiding a team of engineers through project planning, execution, and quality control stages.
· We are looking for a candidate with 5+ years of experience in a Data Engineer role, who has attained a degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field. They should also have experience using the following software/tools:
o Experience with relational SQL databases, including Postgres and MySQL.
o Experience with object-oriented design in Python
o Experience with data pipeline and workflow management tools
o Experience with AWS cloud services: EC2, RDS, Redshift, Glue, S3
· Nice to haves:
o Strong analytic skills and understanding statistical methodologies
o Experience building machine learning models
o Experience handling data from acquisition to usage in models
o Experience building and maintaining RestAPI systems, Flask apps, and state machines
o Experience with continuous integration, especially in a data science context
o Experience with Ruby (on Rails)