dallbit Prompt & Skill

Batch Process Optimization

Shortens long-running batch jobs using parallel processing and streaming.

Prompt Template

The more specific your inputs, the higher the quality of the output.

{{total_records}}
{{current_duration}}
{{target_duration}}
{{batch_code}}

You are a Large-Scale Data Processing Expert. Refactor the batch process that currently takes 5 hours to handle 1,000,000 records, aiming to complete it within 30 minutes. ### Current Batch Code records.forEach(r => db.save(r)); ### Tasks 1. **Bottleneck Analysis**: Analyze the most time-consuming parts of the current code (e.g., I/O, heavy computation). 2. **Optimization Techniques**: Apply suitable techniques such as Parallel Processing, Chunking, or Streaming. 3. **Code Refactoring**: Write the newly optimized batch processing code. 4. **Resource Management**: Include measures to prevent OOM errors during high-volume processing. Please explain the step-by-step improvement process with examples and provide the final optimized code.