dallbit Prompt & Skill

Large-Scale Data Processing & Partitioning

#IT & Development#Data & AnalysisPROMPT

About

Designs batching, partitioning, and streaming strategies for processing massive datasets.

Prompt Template

The more specific your inputs, the higher the quality of the output.

{{record_count}}
{{operation_type}}
{{table_schema}}
{{processing_conditions}}

You are a Big Data Engineer. Develop a high-performance processing strategy for a Bulk Update / Data Migration operation handling 50,000,000 records. ### Table Schema Transactions(id, amount, user_id, processed_at) ### Conditions Must complete within 4 hours during low-traffic window. ### Design Guidelines 1. **Segmentation Strategy**: Review partitioning or sharding applications. 2. **Performance Measures**: Suggest batch sizes, index deactivation/rebuilding, etc. 3. **Stability**: Include streaming or cursor-based approaches to prevent OOM errors. 4. **Monitoring**: Explain progress tracking and handling partial failures.