Selenium
Aws
Docker

Project information

Data Collection Pipeline

  • Developed a module that scraped data from various sources using
  • Curated a database with information about amazon website and stored it on an AWS RDS database using SQLAlchemy and PostgreSQL
  • Performed unit testing and integration testing on the application to ensure that the package published to Pypi is working as expected
  • Used Docker to containerise the application and deployed it to an EC2 instance
  • Set up a CI/CD pipeline using GitHub Actions to push a new Docker image
  • Monitored the container using Prometheus and created dashboards to visualise those metrics using Grafana