We are looking for a Data Engineer or Data Scientist for a 6 month mission with extensions in Brussels. You will be developping a platform that will centralise data on energy efficiency.
Technical context
The platform is set up on Amazon Web Services and uses various PaaS and IaaS solutions. The data lake / Data Warehouse that is used for all kinds of reporting and analysis purposes uses the following Amazon Web services
- Data pipelines and EMR for loading and processing data.
- S3 for saving files
- Amazon Redshift (DWH)
- Tableau as a reporting environment
- Athena for querying flat files in the data lake
- SQL Server as backend database of the Web application
- Lambda for running backend services
- DMS for synchronization of data e.g. from SQL Server to redshift
For example, for processing the consumption data that is supplied on a daily basis, Python and Spark is used as an ETL process to merge the new consumption data in the Redshift DWH with already existing data.
In addition, links are made with numerous other data sources to enrich the Data Warehouse.
GIT is used to Version Control the code and the release process (dev, stg, prod) uses MS DevOps. The latter is also used for following up sprints, user stories, tasks, ...
Your profile
- You manage the existing stack of data flows and pipelines in the AWS cloud environment that uses S3, EMR, Spark, Redshift, Lambda, Python, among others.
- You will (further) build complex data models to generate further analytical insight
- You write high-quality code to further expand the data platform. The platform is scalable and easy to maintain.
- You will link / integrate new data sources from various origins into the existing solution.
- You analyze existing processes and advise where performance gains and cost savings can be achieved.
- You implement data quality controls and alert stakeholders so that the necessary actions can be taken.
- All projects are developed according to the Agile Scrum methodology. As a data engineer you work closely with the other members of the multidisciplinary team.
- You report to the Project Manager of the team.
Interested to know more?
Then apply with your latest CV via the button below!
