Mansur Can

About

Having worked for various projects, I am always looking for a next challenge.

Big Data Engineer & Machine Learning Enthusiast

I am dedicated to navigating the intricate landscape of data management through the lens of the Common Data Model (CDM) project. With a passion for transforming raw data into meaningful insights, I specialize in the seamless mapping of diverse data sources onto the robust framework of the CDM. My journey involves harnessing the power of Azure Synapse Studio and Azure Data Lake to orchestrate this transformation, ensuring precision and efficiency every step of the way. As an advocate for structured and standardized data, I am driven by the vision of unlocking the true potential of information. Join me on this expedition as I bridge the gap between raw data and the enriched possibilities that lie within the Dataverse, creating a harmonious synergy between technology and data-driven innovation.

In my previous involvement, I played a key role in a collaborative migration project where we skillfully transferred data from an RDBMS to HDFS within the Cloudera Data Platform using NiFi, seamlessly bridging the realms of on-premises and cloud environments. Particularly, I led the design of a Spark streaming architecture to capture real-time data from Kafka via Flask, a result of web scraping. This data was carefully stored in HDFS and smoothly integrated with Hive for comprehensive analysis, with the subsequent insights elegantly visualized through Superset, fostering an accessible connection to our Hive repository. This diverse journey showcases my ability to orchestrate data transformation across different platforms, consistently yielding actionable insights.

Freelance: Available
Remote: Available
Location: London, UK

Skills

I have developed a wide range of skills and abilities thorough my studies and work.

Operating Systems	MacOS, Windows, UNIX/Linux
Distributions	Cloudera, AWS, Azure, GCP
Environment	Anaconda, Docker, GitHub, GitLab, Databricks
Database	PostgreSQL, DynamoDB, MongoDB, AWS (RDS, S3)
Scripting	Python, Pyspark, Scala, Java, R, MatLab, SQL, HiveQL, HTML
Data Processing	Apache Spark, Spark Streaming, Pandas, NumPy
Web&App Servers	Goggle Firebase, Flask, Selenium, HTML, CSS, JavaScript
Testing Tools	Unit test, Pytest
Cloud Services	Amazon AWS (EC2, S3, RDS, EMR, DynamoDB, Redshift, CloudFormation); Microsoft Azure (Synapse, Data Factory, Blob, Data Lake, Databricks, DevOps, Power BI, Power Apps); GCP
Hadoop Ecosystem	Hadoop, Cloudera, HDFS, Hive, Kafka, HBase, Cassandra, Zookeeper, Sqoop, Spark, Spark performance Tuning/Optimization, YARN, HBase, Nifi, Oozie, Impala

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. With Amazon EC2, you can launch virtual servers, called "instances", on the cloud. You can choose from various types of instances, each with different combinations of CPU, memory, storage, and networking capacity, to suit your specific needs. You can also customize the operating system, network, and security settings of your instances.

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This makes it a great choice for storing and backing up large amounts of data, such as photos, videos, log files, and more.

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easier to set up, operate, and scale a relational database in the cloud. Amazon RDS supports a variety of popular database engines, including MySQL, MariaDB, PostgreSQL, Oracle, and Microsoft SQL Server. You can choose the engine that best fits your needs, and Amazon RDS will handle the underlying infrastructure and database management tasks for you.

Amazon Elastic MapReduce (EMR) is a web service that makes it easy to process large amounts of data quickly and cost-effectively. It is based on Apache Hadoop, an open-source framework for distributed storage and processing of large datasets. EMR allows you to run a variety of big data frameworks, such as Apache Spark, Apache Hive, and Presto, on a managed cluster of Amazon Elastic Compute Cloud (EC2) instances.

Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse service that makes it simple and cost-effective to analyse all your data using SQL and your existing business intelligence tools. Redshift can be used for a variety of purposes, including analysing data from websites, applications, and other sources to gain insights and make data-driven decisions. It can also be used for reporting and analysis, data lake integration, and data warehousing.

Amazon Web Services (AWS) CloudFormation is a service that helps you automate the process of creating, updating, and deleting cloud resources in an AWS environment. It provides a common language for you to describe and provision all the infrastructure resources in your cloud environment. With CloudFormation, you can use templates written in JSON or YAML to define the resources and their properties in your cloud environment.

Amazon CloudWatch is a monitoring service for AWS resources and the applications you run on the Amazon Web Services (AWS) cloud. It provides data and operational insights for various resource types in the AWS ecosystem, including Amazon Elastic Compute Cloud (Amazon EC2) instances, Amazon Relational Database Service (Amazon RDS) databases, and Amazon Simple Queue Service (Amazon SQS) queues, among others. With Amazon CloudWatch, you can collect and track metrics, set alarms, and automatically react to changes in your AWS resources. You can use CloudWatch to gain system-wide visibility into resource utilisation, application performance, and operational health.

Azure Data Factory is a cloud-based data integration service that allows you to create data-driven workflows in the cloud for orchestrating and automating data movement and data transformation. You can use Data Factory to extract data from various data stores, transform the data based on business needs, and then load the data into other data stores or data lakes for further analysis and reporting.
Data Factory supports a wide range of data stores and formats, including on-premises and cloud-based data sources such as Azure SQL Database, Azure SQL Data Warehouse, Azure Cosmos DB, Azure Blob Storage, Azure Data Lake Storage, and more. It also supports data integration with popular big data and data warehousing technologies such as Apache Spark, Azure HDInsight, Azure Databricks, and Azure Synapse Analytics.

Azure Data Lake is a fully managed, cloud-based data storage and analytics service that enables you to capture data of any size, type, and ingestion speed in one place for operational and exploratory analytics. It is designed to handle a wide variety of data from sources such as applications, websites, IoT devices, and more, and can store data in its original format without requiring transformation or pre-processing.
Data Lake is part of the Azure cloud computing platform, which means you can leverage the scalability, reliability, and security of Azure while using Data Lake to store and analyze your data. Data Lake is built on top of Azure Storage, which is a highly scalable and durable storage platform that can store and process petabyte-scale datasets.

Azure Databricks is a fully managed, cloud-based data analytics service that allows you to build, train, and deploy data analytics and machine learning models quickly and easily. It is built on top of the popular open-source data analytics platform Apache Spark and offers a range of features and capabilities that make it easier to work with big data in the cloud. With Azure Databricks, you can create notebooks, which are interactive documents that contain code, text, and other elements, and use them to explore, transform, and visualize data.

Azure Synapse Analytics (formerly SQL Data Warehouse) is a cloud-based data warehouse service that integrates seamlessy with Azure Big Data and Azure Machine Learning. It allows you to create a data warehouse that can store and process data using SQL and Spark, and it can be easily integrated with other Azure services such as Azure Data Lake Storage and Azure Databricks.
With Azure Synapse Analytics, you can analyze large amounts of data quickly and easily using SQL and Spark, and you can use the service to build data pipelines and create data lakes for storing and processing large volumes of data. It also provides a variety of tools and features for visualizing and exploring data, as well as for integrating with other Azure services and tools.

Azure Blob storage is a cloud-based object storage service provided by Microsoft Azure. It allows you to store large amounts of unstructured data, such as text or binary data, in a cost-effective and scalable way.
Blob storage is optimised for storing massive amounts of unstructured data, such as videos, audio, images, and documents. Blob storage is a popular choice for storing and serving large amounts of data, and it is often used in conjunction with other Azure services, such as Azure Synapse Analytics, Azure Data Lake Storage, and Azure Databricks, to build data pipelines, data lakes, and other big data solutions. It is also frequently used to store and serve static assets, such as images, videos, and documents, for websites and applications.

Microsoft Azure Power BI is a cloud-based business analytics service that enables you to visualize, analyze, and share data insights. It allows you to connect to a wide range of data sources, both on-premises and in the cloud, and to create interactive, visually appealing reports and dashboards. Power BI is designed to be easy to use and accessible to users of all skill levels, and it provides a wide range of features and tools for data exploration, visualization, and collaboration.

Cloudera Data Platform (CDP) is a cloud-based data management platform developed by Cloudera, a company that provides a distribution of Apache Hadoop, an open-source software framework for storing and processing large amounts of data in a distributed computing environment. CDP is designed to be a comprehensive platform for data management, data processing, and data analytics, and it includes a wide range of tools and features for working with big data. CDP is built on top of Cloudera's distribution of Hadoop, called Cloudera Enterprise, and it includes all of the core components of Hadoop, as well as a variety of additional tools and features for data management, data processing, and data analysis. It is highly scalable and flexible, and it can be used to build a wide range of data-driven applications, including data warehouses, data lakes, and real-time streaming applications.

Python is a popular, high-level programming language known for its simplicity, readability, and flexibility. It was created in the late 1980s by Guido van Rossum, and it has a large and active community of users and developers. One of the key strengths of Python is its simplicity and ease of use. The language has a relatively small number of keywords, a clear and concise syntax, and strong readability, which makes it a good choice for beginners and experienced programmers alike. Python is a general-purpose programming language, which means that it can be used to build almost any type of software, from web applications to desktop applications to scientific applications and more.

Apache Spark is an open-source, distributed computing system designed for fast processing of large datasets. It is a key component of the Apache Hadoop ecosystem, and it is widely used for big data processing and analytics. Spark is known for its speed, as it can process data much faster than traditional Hadoop MapReduce. It achieves this speed through the use of in-memory computing, which allows it to store and process data in memory rather than reading and writing it to disk. This makes Spark well-suited for real-time data processing and interactive data analytics. In addition to its speed, Spark is also known for its versatility and ease of use. It provides a wide range of APIs for different programming languages, including Python, R, Java, and Scala, and it has a number of libraries and tools for data processing, machine learning, and graph processing.

Scala is a programming language designed to be expressive, concise, and statically typed. It was created in the early 2000s by Martin Odersky and has since gained a strong following among developers. Scala is a general-purpose programming language that can be used to build a wide range of applications, from web applications to data analysis tools to mobile apps and more. It is often used in combination with other technologies, such as the Apache Spark distributed computing system, to build scalable, high-performance applications. One of the key strengths of Scala is its support for functional programming, which allows developers to write code in a more declarative and expressive style. It also has a strong type system, which helps to catch errors at compile-time rather than runtime, and it has good support for concurrency and parallelism, which makes it well-suited for building distributed systems.

Apache Kafka is an open-source distributed event streaming platform used for building real-time data pipelines and streaming applications. It was created at LinkedIn and has since become one of the most popular event streaming platforms in the world, with a large and active community of users and developers. Kafka is designed to be scalable, fault-tolerant, and high-performance, and it is often used to build real-time data pipelines that can handle millions of events per second. It is also used for building real-time streaming applications, such as real-time analytics, fraud detection, and social media feeds. At its core, Kafka is a message queue that allows producers to send messages to one or more topics and allows consumers to read messages from those topics. It is highly distributed and fault-tolerant, and it is designed to allow messages to be processed asynchronously, which makes it well-suited for building distributed systems.

The Hadoop ecosystem refers to the collection of open-source technologies that are built on top of or alongside Apache Hadoop, an open-source software framework for storing and processing large amounts of data in a distributed computing environment. The Hadoop ecosystem includes a wide range of tools and technologies that are designed to work together to enable organizations to store, process, and analyze large amounts of data at scale. Some of the key components of the Hadoop ecosystem include: HDFS, Yarn, MapReduce, NiFi, Imapala, Hive, Hue, Sqoop, HBase, Cassandra, Zookeper.

Resume

I have a strong background in research and have worked on a wide variety of projects.

Education

Specialist AI and Data School Program

AI Core

Certified in the practical application of AI & Data Engineering using industry-standard tools including:

Computer Vision Project
Trained a computer vision model and used Tensorflow to detect whether Rock, Paper or Scissors is shown to the camera in real-time and with a high accuracy. Used the OpenCV library to access the webcam and play Rock Paper Scissors with the computer using the image from the camera.

Data Collection Pipeline Project
Develope a module that scraped data from various sources using Selenium Curated a database with information about and stored it on an AWS RDS database using SQLAlchemy and PostgreSQL Performed unit testing and integration testing on the application to ensure that the package published to Pypi is working as expected Used Docker to containerize the application and deployed it to an EC2 instance Set up a CI/CD pipeline using GitHub Actions to push a new Docker image.

PhD in Physics

Cardiff University, UK

Throughout a year and a half of my PhD project, I concentrated on nanoparticle characterization through a combination of optical microscopy, image analysis, and advanced simulation techniques. My utilization of Comsol and MatLab played a pivotal role in the exploration of intricate nanoparticle behaviors, enabling accurate modeling and simulation of complex interactions at the nanoscale. This integration of Comsol and MatLab enriched my research, enhancing the depth and precision of nanoparticle analysis and characterization.

MSc in Software Engineering

Westmister University, UK

Dissertation: Learning model for handwritten digit recognition - Convolutional Neural Network (CNN) model was created in Python and run on Swift IOS application
Research Methods and Professional Practice
Big Data Theory and Practice
Mining and Machine Learning (R)
R-5-week-study-tutorials
Enterprise Development (C#)
Advanced Software Design (UML)
Mobile Application Development (Swift-XCode)
Concurrency and Parallelism (Java)

Postgraduate Diploma in Biomedical Engineering

Brunel University, UK

3 Electrode Graphene on Paper Biosensors for DNA Test project
Software programs were used in projects are CFD, Abaqus, MatLab

PGCE (Secondary) Science

Goldsmiths, University of London, UK

Postgraduate Certificate in Nanotechnology

University of Oxford, UK

BSc in Physics

Karadeniz Technical University, Turkey

I studied Pascal programming language during my undergraduate degree.

Portfolio

The projects demonstrate my ability to solve problems, write clean and efficient code, and work on a team. It is important to have a diverse range of projects that showcase my abilities and highlight my unique strengths as a software engineer.

All
App
Web
Program

Combined Four Countries

Pyspark

Facial Recognition IOS Application

Swift and Python

Computer Vision

Open CV Teachable Machine

Learning Model for Handwritten Digit Recognition

CoreML Tensorflow Keras

Turkish Tutor Website

Bootstrap Html Css

Webscraper

Selenium Aws Docker

Portfolio Resume Website