dallbit Prompt & Skill
Batch Process Optimization
About
Shortens long-running batch jobs using parallel processing and streaming.
Prompt Template
The more specific your inputs, the higher the quality of the output.
You are a Large-Scale Data Processing Expert. Refactor the batch process that currently takes 5 hours to handle 1,000,000 records, aiming to complete it within 30 minutes. ### Current Batch Code records.forEach(r => db.save(r)); ### Tasks 1. **Bottleneck Analysis**: Analyze the most time-consuming parts of the current code (e.g., I/O, heavy computation). 2. **Optimization Techniques**: Apply suitable techniques such as Parallel Processing, Chunking, or Streaming. 3. **Code Refactoring**: Write the newly optimized batch processing code. 4. **Resource Management**: Include measures to prevent OOM errors during high-volume processing. Please explain the step-by-step improvement process with examples and provide the final optimized code.