Real-time Stream Processing Using Kafka And Spark

$ 30.00 — 250.00 USD

Hi all,

I have a Real-time stream processing project using Kafka and Spark
I need someone who is good in Python, PySpark, Kafka and Jupyter Notebook
I have done some of the work on this project (like 45%) and I need someone to finish it for me.

NOTE: IT IS EXTREMLY IMPORTANT THAT YOU USE JUPYTER NOTEBOOK AND THE TEMPLATES PROVIDED

● Task 1 (Already Done): Kafka Producer - Producing the streaming data, where you can use csv modules to read and publish the data to the Kafka stream.

● Task 2 (Partially Done): Kafka Consumer - Consuming the streaming data using Kafka consumer, use the csv module in any python libraries (e.g.
Pandas) to process the ingested data from Kafka.

● Task 3 (Not attempted): Streaming application - Using Spark Structured Streaming together with Spark ML/SQL to process data streams.
In task 3, pandas can only be used for plotting steps.

The excessive usage of Pandas for data processing is not recommended.

In this task, we will implement Spark Structured Streaming to consume the data from task 1 and perform streaming classification.
You may use Spark Structured Streaming together with Spark SQL and ML.
You are also provided with a set of pre-trained pipeline models to classify the binary classification flight-delays.

NOTE: PLEASE LABEL THE STEPS IN TASK 3 AS THEY CORRESPOD TO THE INSTRCTIONS FILE

You can find the rest of the instructions in the file (Instructions.pdf)

There are three ipynb files:
1.
Task1_flight_producer.ipynb
2.
Task2_flight_consumer.ipynb
3.
Task3_streaming_application.ipynb

The dataset for this project consists of 20 csv flight files (flight*.csv) and the metadata.pdf and airports.csv.
You only need to use flight*.csv files.
The file is included in the attachments (flight-delays.zip)
The model to be used in Task 3 is also included in the attachments (model.zip)

The Timeline for this project is 2 days.
Please let me know if you can do it in this time or not

PythonMachine Learning (ML)PySparkApache KafkaApache Spark

Similar Freelance jobs:

Write Some Software

Design, Media & ArchitectureSales & MarketingPythonSoftware ArchitectureC++ Programming Place Bid

Full Description of Write some Software

Hardware Applications

Design, Media & ArchitecturePythonCSSHTMLArduino Place Bid

Full Description of Hardware applications

Machine Learning - Ninjatrader -- 2

C# ProgrammingMachine Learning (ML)Statistical AnalysisArtificial IntelligenceTrading Place Bid

I need a Machine Learning Engine to Optimize entry and and exit . I have programmed a custom indicator for the Ninatrader platform that will output different up to 16 variables (Characters) from a set of 6 possible outcomes. To simplify I have made them Characters, ABCXYZ.. In a series (look back period) of up to 10.. Example of one series of a look back of 10. There will be 8 of these. ZZZZYYXYXX. In essence I would like to…

Full Description of Machine Learning - Ninjatrader -- 2

Zamaso Inc

Design, Media & ArchitecturePythonHTMLcURLBash Scripting Place Bid

I need scripts from textnow or any wesbsite of sending free sms With this script i need to send bulk sms

Full Description of Zamaso inc

I Want To Build A Tool Using Hap.py Benchmarking (nist) To Validate Any Vcf File By Running The File

JavaMobile Phones & ComputingPythonSoftware ArchitectureBiotechnology Place Bid

The inputs to hap.py are two VCF files (a "truth" and a "query" file), the truth file is found here https://www.nature.com/articles/sdata201625/tables/3 which includes extensive genome sequencing for 1 person. So we can validate any VCF file by comparing our VCF with the truth VCF and the outcome will include a table with TP FP TN FN and precision with the sensitivity you can read more about it on this page: https://github.com/RealTimeGenomics/rtg-tools https://github.com/Illumina/hap.py/blob/master/doc/happy.md#working-with-genome-vcfs

Full Description of I want to build a tool using Hap.py…

Api & Web App For Interactive Brokers

Design, Media & ArchitecturePythonWeb ScrapingC# ProgrammingSoftware Architecture Place Bid

I need an API & web app for Interactive brokers that will buy and sale stocks based on settable criteria in the API. The trades will be based on stock information received from a 3rd party data scraping service. The API must be able to trade three separate types of signals. See attached notes and UI layout for details of the project.

Full Description of API & Web App for Interactive Brokers

Secure Link Topology Architecture For Software Defined Networking (sdn)

PythonSoftware ArchitectureMininetRyu Controller Place Bid

I'm a researcher in Computer Networks, whose areas of interest in Software Defined Networks. 1.The project is to re-implement the attached file on SLDP: A secure and lightweight link discovery protocol in software defined networking and get the actual results with the use of Mininet emulator and Ryu controller. Also identify the weaknesses of the work 2. After the re-implementation, the weaknesses are to be strengthen 3. propose a secure and distributed link discovery protocol for software defined networking 4.…

Full Description of Secure Link Topology Architecture for Software Defined Networking…

Build An Sram Memory Compiler Based On "openram" Project.

PythonElectrical Engineering Place Bid

A configurable SRAM compiler written in Python to tile the bit cells and address decoder blocks, extract the Verilog and parasitic RC models, and generate the GDS2 block database. The project should leverage the "openRAM" open source code as much as possible to achieve the above objectives.

Full Description of Build an SRAM memory compiler based on "openRAM"…

Create A Script To Collect Information -- 2

Design, Media & ArchitecturePythonAutomation Place Bid

I would like to create a script to automate some repettive task on Excel and to collect some info on Google and a Website

Full Description of Create a script to collect information -- 2

Develop A Profitable Crypto Bitcoin Mining Bot With Api -- 2

Design, Media & ArchitectureMobile Phones & ComputingPythonSoftware ArchitectureMySQL Place Bid

DEVELOP A PROFITABLE CRYPTO BITCOIN MINING BOT WITH GREAT Features+ Secured Admin Access+ Api Features

Full Description of DEVELOP A PROFITABLE CRYPTO BITCOIN MINING BOT WITH…

Statistical Tts & Stt Desktop Softwar Engine Which Will Work With Jaws

PythonSoftware ArchitectureSoftware TestingVoice TalentC++ Programming Place Bid

Desktop voice is desktop-based software that is installed as X-language voice on a computer with Windows operating system. This software engine after installation is used by the operating system to synthesize words in X-Language. This software engine can be used by other application programs such as voice, e.g. in JAWS screen reader this software/voice can be selected so that the words synthesized by JAWS are synthesized in X-Language. Also, this voice can be used by other programs that require the…

Full Description of Statistical TTS & STT Desktop softwar engine which…

Python Program

Python Place Bid

It is a project for RPA (Robotic Process Automation).

Full Description of Python Program

Looking For Experience Sms Platform Devleoper

Design, Media & ArchitecturePythonDjangoFreelancer API Place Bid

Looking for an experienced GUI developer for the Jasmin SMS gateway portal.

Full Description of Looking For Experience SMS Platform Devleoper

Maquetador En Odoo

PythonWebsite DesignERPOdooWeb Development Place Bid

Se Busca desarrollador en Odoo con experiencia maqueando paginas desde cero. Maquetar paginas del sitio web. Maquetar plantillas de correo electrónico

Full Description of Maquetador en odoo

Experienced Python Coder Needed

PythonSoftware Architecture Place Bid

I need the python coder for multiple jobs. Details will be discussed.

Full Description of Experienced Python coder needed