Need some assistance with crawler, A professional developer or programmer help would really helps

Hello,

I need help with something really hard, So first of all I am not a hacker or something close to that, but I am capable of quite a lot of stuff, Now I am trying to find something on tor network, something that is super hard to find, For the same I did some crawling, So I don't know much about coding and programming but I somehow built myself a crawler with use of AI, now I ran it discovered a lot of sites but unable to find what I was trying to find.

So, I am trying to make a new crawler but the issue is I don't know programming as mentioned earlier and I don't have the time and patience to learn python, scrapy and other stuff and then learn more stuff, I tried my best to work with AI but AI is keep failing now I need some real human support and help.

So let me mention all the features I want in the new crawler and I have a code that I generated from AI, If you want I can share the code with you but that code contain a lot of issues, making something new from scratch would be more useful and time saving.

The features I want is -

Core Engine

High-Concurrency Engine: Built on asyncio to run 1000 workers at once.

Redis Job Queuing: Uses Redis to manage the to-do list for all workers.

Fault-Tolerant Job Recovery: Automatically re-queues jobs from any worker that crashes, so nothing is lost.

Automatic Retry System: Retries failed URLs 5 times before giving up on them.

Graceful Shutdown: Cleans up all connections and finishes current jobs when you press Ctrl+C.

Network & Evasion

Rotating Proxy Pool: Manages and rotates through a list of 200 SOCKS proxies.

Browser Impersonation: Uses httpx-curl-cffi to make requests look exactly like they're from a Chrome 110 browser.

Realistic Stealth Headers: Sends a full set of rotating, realistic browser headers to avoid detection.

Intelligence & Data

Contact Discovery: Scrapes pages to find and save emails, Telegram handles, and XMPP IDs.

Site Requirement Analysis: Detects keywords to see if a site requires a login, captcha, or registration.

Keyword Frequency Analysis: Ranks the most common keywords found across all sites, ignoring junk words.

Advanced Link Extractor: Finds all new links on a page, even if they are just plain text and not in <a> tags.

Autonomous Discovery Mode: Automatically takes the most popular keywords found and uses them to search for new sites on its own.

Tools & Utilities

Live Terminal Dashboard: A heads-up display showing crawl speed (URLs/min), active workers, and proxy health.

PostgreSQL Database: Stores all final, organized intelligence.

CSV/JSON Data Export: A script to dump all the collected intelligence from the database into a simple file.

Keyword List Export: A script to save the top-ranked keywords from Redis to a text file.

Database Initializer: A command (--init-db) to create all the necessary database tables for you.

URL Seeder: A command (--seed-file) to load a starting list of URLs from a text file.

There is more but I think we can discuss that later, Now let me mention my previous crawler issues ok -

1. The Critical "Missing Scheduler" Bug

This is the main reason your crawler isn't working. It's a show-stopper.

What's Happening: The crawler finds new URLs (either from the SearchManager or from scraping a page) and correctly saves them to the PostgreSQL database.

The Problem: It never adds those new URLs to the Redis job queue (crawler_frontier_queue).

The Result: The 1000 workers are all sitting idle, waiting for jobs to appear in the Redis queue. But since no URLs are ever put there, they have nothing to do. The only part of your program that runs is the SearchManager, which just finds URLs and puts them in the database, where they sit forever.

2. Worker vs. Proxy Mismatch

This is a major resource and logic error.

What's Happening: You have num_workers: 1000 set in your configuration file.

The Problem: You only have 200 socks_ports (proxies) defined.

The Result: 800 of your workers will start, instantly try to get a proxy, fail, and sit there asleep, waiting for a proxy to become available. You are wasting resources running 800 workers that can literally never do any work. Your actual number of workers is 200, not 1000.

3. Wasting Time Retrying 404s

This is a logical flaw in the worker.

What's Happening: The worker code uses response.raise_for_status() to check if a crawl was successful.

The Problem: This function treats errors like "404 Not Found" or "403 Forbidden" as failures. These failures are then sent to the _failure_processor, which re-queues them for a retry.

The Result: Your crawler will waste time and proxies trying to crawl a non-existent page 5 times before it gives up. A "404" is a successful crawl (we learned the page isn't there) and should not be retried.

4. Complex Manual Proxy Setup

This is a setup-killer.

What's Happening: The generate_tor_config.py script just prints a 200-line configuration file to your terminal.

The Problem: It doesn't actually do anything. You are expected to manually copy that text, log into your proxy server, edit the system's torrc file, create 200 separate data directories, and then restart the proxy service.

The Result: If you just ran this script and thought your proxies were set up, they are not. The crawler has no proxies to connect to and will fail on every single job.

5. Brittle curl-cffi Dependency

This is a small but annoying setup issue.

What's Happening: The requirements.txt file includes httpx-curl-cffi.

The Problem: This library requires the curl system library to be installed on your machine.

The Result: If it's not installed, or if it's the wrong version, the entire crawler will crash on startup.

The Critical "Missing Scheduler" Bug

This is the main reason your crawler isn't working. It's a show-stopper.

What's Happening: The crawler finds new URLs (either from the SearchManager or from scraping a page) and correctly saves them to the PostgreSQL database.

The Problem: It never adds those new URLs to the Redis job queue (crawler_frontier_queue).

The Result: The 1000 workers are all sitting idle, waiting for jobs to appear in the Redis queue. But since no URLs are ever put there, they have nothing to do. The only part of your program that runs is the SearchManager, which just finds URLs and puts them in the database, where they sit forever.

2. Worker vs. Proxy Mismatch

This is a major resource and logic error.

What's Happening: You have num_workers: 1000 set in your configuration file.

The Problem: You only have 200 socks_ports (proxies) defined.

The Result: 800 of your workers will start, instantly try to get a proxy, fail, and sit there asleep, waiting for a proxy to become available. You are wasting resources running 800 workers that can literally never do any work. Your actual number of workers is 200, not 1000.

3. Wasting Time Retrying 404s

This is a logical flaw in the worker.

What's Happening: The worker code uses response.raise_for_status() to check if a crawl was successful.

The Problem: This function treats errors like "404 Not Found" or "403 Forbidden" as failures. These failures are then sent to the _failure_processor, which re-queues them for a retry.

The Result: Your crawler will waste time and proxies trying to crawl a non-existent page 5 times before it gives up. A "404" is a successful crawl (we learned the page isn't there) and should not be retried.

4. Complex Manual Proxy Setup

This is a setup-killer.

What's Happening: The generate_tor_config.py script just prints a 200-line configuration file to your terminal.

The Problem: It doesn't actually do anything. You are expected to manually copy that text, log into your proxy server, edit the system's torrc file, create 200 separate data directories, and then restart the proxy service.

The Result: If you just ran this script and thought your proxies were set up, they are not. The crawler has no proxies to connect to and will fail on every single job.

5. Brittle curl-cffi Dependency

This is a small but annoying setup issue.

What's Happening: The requirements.txt file includes httpx-curl-cffi.

The Problem: This library requires the curl system library to be installed on your machine.

The Result: If it's not installed, or if it's the wrong version, the entire crawler will crash on startup.

Now lets talk about my experience and other things, Now I done some crawling by myself, I got the crawler coded by AI a different crawler not a full fledge crawler more like a script ok, then ran it in VM whonix and it was quite slow crawler as it was only working with 5 workers, and I discovered more than 10million links and I was still unable to find what I was looking for, Now as it was only a script that was not doing much, Me finding what I am trying to find is like no near game with that crawler, For the same I try to make a new crawler but AI is not able to help me with this.

For the same I am here.

Now I don't only want the crawler code or a full fledged crawler, I want your help to find me what I am trying to find, And like you can run it and tell me or share all the results with me.

I know what I am talking about, How hard it is, How deep it is and everything else, I am ready to take all the necessary steps in orders to get the whole thing working, I am willing and ready to work with you.

I hope you understand the whole thing, and find me what I am trying to find and if you don't do this, find me a person who can do this and I hope so everything would pay off big.