The Synergy of Automated Pipelines with Prompt Engineering and Generative AI in Web Crawling