IoT_Devices_To_Kafka_Using_Scala_Spark

₹ 1,500.00 — 12,500.00 INR
Existing Flow:
Iot Devices(Sending Json Payload) -> Kafka Streaming(receiving & Producing) -> Spark Streaming(Consuming Json using Scala) -> Cassandra(For Storage)

Project Description:
IoT devices getting uploaded with JSON payload in Cassandra.

IoT devices sending device information through a simulator to Kafka streaming.

Kafka Producer producing JSON streaming data.

Kafka JSON data-consuming in Scala 2.11.11 with Spark 2.4.7
Spark consumed data getting inserted into Cassandra.

Requirement:
1.
Performance Tuning
-> Need to persist @100/sec JSON payload into Cassandra using spark streaming

2.
Project should support higher version of Cassandra Spark Connector with latest Spark and Scala
(Need to validate code on higher version compatibility)

Note: -> We've already achieved @40/sec.
We'll share it.
-> Project deployed on Linux( RAM 16 GB, 8 CPU Cores)
-> We tested on Spark 2.4.7, Scala 2.11.11, and Spark Cassandra Connector 2.4.0

As per the required skillset and interest will share the code.

Similar Freelance jobs:

Problem With Spark Ar Filter
Need filter for instagram to be fixed. The image should be inside but instead it is cut off as seen in the image
Full Description of problem with Spark AR filter
Spark Scla
Expertise in designing and deployment of Hadoop Cluster and different analytical tools including Pig, Hive, HBase, Sqoop, Kafka Spark with Cloudera distribution. Working on a live 20 nodes Hadoop cluster running on CDH4.4. Working with highly unstructured and semi structured data of 40 TB in size (120 TB with replication factor of 3) Managing external tables in Hive for optimized performance. Very good understanding of Partitions and Bucketing in Hive Developed Spark scripts using Scala as per the requirement using…
Full Description of Spark Scla
Scala Developer Required With Big Table
Hi All, I am looking for scala developer with below knowledge; SCALA, BIGTABLE BIGQUERY Regards Pavan
Full Description of Scala developer required with big table
Bash Script- Automation
this project requires a script to do the following 1. gather directories and subdirectores to depth of 5 and fetch the checksum values assign them to variables 2.recurssivley verify directories, subdirectories and files if there is any change from previous ( ex: checksum ) 3. consider no change variables as nodiff and changed variable values as diff 4. use sed or other command to alter the values of nodiff variables recursivley to ! 5. we should have a preview…
Full Description of Bash script- automation
Distributed And Scalable Big Data Project (apache Spark ,apache Nifi, Cassandra, Elasticsearch)
We look for two big data developers to work on developing: - distributed Cassandra (huge data entry) , process data with spark(enhance parallel computing), a search engine( Elasticsearch) - simple graphic user interface.
Full Description of Distributed and scalable Big Data Project (Apache Spark…
Big Data Project
Need: We are looking to form a team of two skilled developers that are willing to work on a big data project for the three different tasks that we will explain in a bit. Objective: The objective is to scale the platform for future needs as well as address limitations of retrieving data by building a mini search engine with Elasticsearch. Minimizing the cost is important factor which motivated the use of open-source tools such as: Apache NiFi Apache Spark…
Full Description of Big Data project
Python Engineer
Experience in Python is a must. Experience on C# would be an added advantage. ● Proficiency with fundamental front-end languages such as HTML, CSS, and JavaScript, Jquery. ● Familiarity with JavaScript frameworks such as Vuejs, Angular JS, React. ● Beneficial to have working experience on POS Systems or any other hardware embedded & IoT systems. ● Worked on enterprise grade systems. ● Have designed web services (RESTful web services). ● Know how to scale systems that have database bottlenecks etc.…
Full Description of Python Engineer
Mongodb Big Data Python
i have a database on mongodb, and i have multiple tables ,, sometimes i have tables that have points in common, and so i want the table fields to be updated from their common data and which are not filled in their fields
Full Description of Mongodb big data python
B.tech Computer Science Expert
I need someone who are really experienced in these following domains:- -Python -Spark -AWS Glue -SQL -AWS Cloud Formation -Aws Code Pipeline Please If you have real experience then only bid this project. It will be a contact based job. If we will find out you an experienced one and If you satisfy us with your work, we'll keep working for longer period. You will paid hourly for every projects.
Full Description of B.Tech Computer Science Expert
Big Data Spark, Hadoop
Big Data Spark, Hadoop Find a dataset. Your dataset should contain at least 10,000 records. Select a business problem that can be addressed through machine learning.
Full Description of Big Data Spark, Hadoop
© 2006 — 2026 hirelancer.com is an affiliate website, listing the freelance projects. We are collaborating with other sites like getFreelancer.com. Feel free to bid on any project and good luck! You will need register first, before bidding or posting projects. Basic memberhsip is always free. We might earn commissions from referring you. You will never ever pay additional fees for we referred you.