Update program bugs - python script

Job ID: 30637319

Budget: $30 – $250 USD

Milestone 1: The program needs to run as expected. These are some of the issues its facing right now.

Issue 1
Currently I have an API from the PA-API ( Amazon product advertising API) that is suppose to update the .csv file 8-10 ASINs/ second. It currently updates the .csv 8-10 ASINS/ second if the ASINS are being "updated" , however if a new ASIN that we are searching for is input, then it only updates at 1 ASIN/second. This is possibly 2 reasons why its only updating at that slow speed.
1. Its because the previous developer didn't code it properly to check for duplicates. What I need it to do is every time I input a new .csv with new ASINS to check against the api_output.csv and then add any NEW asins that don’t already exist onto the api_output.csv. Then start updating the database. I believe currently it checks for duplicates every time it trys to list a new ASIN ( very inefficient)
2. We should save it to MySQL instead of to a .csv. I need to be able to access MySQL easily.

MySQL is a solution we can implement if you believe it’ll be faster/ more efficient to reach our goal.

Issue 2:
If i turn off the server or disconnect from it and then turn it back on, most of the ASIN data from the MASTER csv are lost for some reason.
1. Solution maybe MySQL? or some database i can access easily from anywhere.

Issue 3:
If the program stops, it doesn't start where it left off. It starts from the beginning. I need it to start where it left off last. Both the API searching and the ebay extraction of data. Also data is lost if i stop it in middle of it running.

Issue 4:

I need to have a filter ( settings) where it can filter each of the headers in the current .csv ouput from ebay.csv.

Issue 5:

Develop a script that will run all the other scripts in order, without me having to run each one after it finishes.

Issue 6:

Index 88 says lowest price is 5.49 from Bestbuy … but really its just not picking up the lowest price b/c it says “see price” there so its not really lowest price… 5.49 is just shipping cost not cost of item.

Milestone 2 : I will provide a MySQL or some other database with which items to search. The MySQL will either have the UPC include so we can search it or we will use the PA-API and/or the Master CSV sheet to obtain the UPC in order to search on ebay ( using the program that exists) for the lowest price, and the store name of the lowest price. This will need to run once every XX hours ( setting to change it). The only field that the current program doesn't extract is the ebay store name, so we will need to update it to pull that data too.
Related categories: Python Web Scraping MySQL Debugging Scrapy