{ "cells": [ { "cell_type": "markdown", "id": "f79bc1c1-ef37-4cef-972a-89e4e6989474", "metadata": {}, "source": [ "# S3 Manifesto Quick Start Tutorial\n", "\n", "This notebook demonstrates how to use s3manifesto for efficient file metadata management and intelligent partitioning in big data workflows.\n", "\n", "## Overview\n", "\n", "s3manifesto provides:\n", "\n", "- **Metadata Organization**: Consolidate file metadata into manageable collections\n", "- **Two-File Storage**: Efficient JSON summary + Parquet data file system \n", "- **Intelligent Partitioning**: Best Fit Decreasing algorithm for optimal file grouping\n", "- **Divide-and-Conquer Ready**: Perfect for distributed processing frameworks\n", "\n", "Let's walk through a complete example!" ] }, { "cell_type": "markdown", "id": "27a35a6e-3812-4897-8306-8936ce5e2bfb", "metadata": {}, "source": [ "## 1. Setup Mock AWS Environment\n", "\n", "For this tutorial, we'll use moto to create a mock AWS environment. In production, you would connect to your actual AWS S3 service." ] }, { "cell_type": "code", "execution_count": 1, "id": "ac4d5de1-3092-4637-ae0e-29506bd66850", "metadata": {}, "outputs": [], "source": [ "import moto\n", "\n", "mock_aws = moto.mock_aws()\n", "mock_aws.start()" ] }, { "cell_type": "markdown", "id": "3c42d68a-6c0e-40cf-9b78-7f151916f6fd", "metadata": {}, "source": [ "## 2. Create S3 Bucket and Client\n", "\n", "Set up our S3 client and create a bucket for storing manifest files." ] }, { "cell_type": "code", "execution_count": 2, "id": "cca1fb74-b84c-4af1-a33e-3a6c99df4044", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "✅ Created S3 bucket: my-bucket\n" ] } ], "source": [ "import boto3\n", "\n", "boto_ses = boto3.Session(region_name=\"us-east-1\")\n", "s3_client = boto_ses.client(\"s3\")\n", "\n", "s3_bucket = \"my-bucket\"\n", "s3_client.create_bucket(Bucket=s3_bucket)\n", "\n", "print(f\"✅ Created S3 bucket: {s3_bucket}\")" ] }, { "cell_type": "markdown", "id": "ab68a683-4454-49e7-9cbc-9ec244ff2501", "metadata": {}, "source": [ "## 3. Create Sample Data Files and Manifest\n", "\n", "Let's create a manifest containing metadata for 10 sample data files. Each file represents a typical big data scenario with varying sizes and record counts.\n", "\n", "### What is a Manifest?\n", "\n", "A manifest is a collection of file metadata containing:\n", "\n", "- **URI**: S3 location of each data file\n", "- **Size**: File size in bytes for resource planning \n", "- **Record Count**: Number of records for workload estimation\n", "- **ETag**: Data integrity verification hash" ] }, { "cell_type": "code", "execution_count": 18, "id": "e54fb8fe-712e-4967-a4c1-ad6e757bbb0c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "📊 Created manifest with 10 files\n", "📏 Total size: 102.92 MB\n", "📦 Total records: 10,036,408\n", "🔐 Fingerprint: 14f938cb78d554e52039879fddb1f1a4\n" ] } ], "source": [ "import s3manifesto.api as s3manifesto\n", "import random\n", "\n", "n_file = 10\n", "uri = f\"s3://{s3_bucket}/manifest.parquet\"\n", "uri_summary = f\"s3://{s3_bucket}/manifest-summary.json\"\n", "\n", "data_file_list = list()\n", "for ith in range(1, 1 + n_file):\n", " data_file_uri = f\"s3://{s3_bucket}/data/{ith}.parquet\"\n", " # 500K ~ 1.5M records each file\n", " n_record = random.randint(500_000, 1_500_000)\n", " # 5MB ~ 15MB size each file\n", " size = random.randint(5 * 1_000_000, 15 * 1_000_000)\n", " data_file = s3manifesto.DataFile(\n", " uri=data_file_uri,\n", " etag=\"...\",\n", " size=size,\n", " n_record=n_record,\n", " )\n", " data_file_list.append(data_file)\n", "\n", "manifest_file = s3manifesto.ManifestFile.new(\n", " uri=uri, # URI for manifest data file (Parquet format)\n", " uri_summary=uri_summary, # URI for manifest summary file (JSON format)\n", " data_file_list=data_file_list,\n", " calculate=True, # Automatically calculate totals from data_file_list\n", ")\n", "\n", "print(f\"📊 Created manifest with {manifest_file.n_data_file} files\")\n", "print(f\"📏 Total size: {manifest_file.size_for_human}\")\n", "print(f\"📦 Total records: {manifest_file.n_record:,}\")\n", "print(f\"🔐 Fingerprint: {manifest_file.fingerprint}\")" ] }, { "cell_type": "markdown", "id": "149b68e8-4302-4b67-b471-1dae52b727f1", "metadata": {}, "source": [ "## 4. Write Manifest to S3\n", "\n", "A manifest consists of **two files** that work together:\n", "\n", "1. **Manifest Summary File** (JSON): Contains aggregate metadata and references to the data file\n", "2. **Manifest Data File** (Parquet): Contains detailed per-file metadata in high-performance format\n", "\n", "This two-file system allows for efficient metadata access - you can quickly check summary statistics without loading the full detailed metadata." ] }, { "cell_type": "code", "execution_count": 21, "id": "84fe7d00-80ca-49ee-b756-a582f5ee1969", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "✅ Written manifest to S3:\n", " 📄 Summary: s3://my-bucket/manifest-summary.json\n", " 📋 Data: s3://my-bucket/manifest.parquet\n" ] } ], "source": [ "manifest_file.write(s3_client=s3_client)\n", "print(f\"✅ Written manifest to S3:\")\n", "print(f\" 📄 Summary: {uri_summary}\")\n", "print(f\" 📋 Data: {uri}\")" ] }, { "cell_type": "code", "execution_count": 22, "id": "016dd59a-af78-4662-a170-d42126f13074", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "content of manifest summary file:\n" ] }, { "data": { "text/html": [ "
{\n",
       "    'manifest': 's3://my-bucket/manifest.parquet',\n",
       "    'size': 107924092,\n",
       "    'n_record': 10036408,\n",
       "    'fingerprint': '14f938cb78d554e52039879fddb1f1a4',\n",
       "    'details': {}\n",
       "}\n",
       "
\n" ], "text/plain": [ "\u001b[1m{\u001b[0m\n", " \u001b[32m'manifest'\u001b[0m: \u001b[32m's3://my-bucket/manifest.parquet'\u001b[0m,\n", " \u001b[32m'size'\u001b[0m: \u001b[1;36m107924092\u001b[0m,\n", " \u001b[32m'n_record'\u001b[0m: \u001b[1;36m10036408\u001b[0m,\n", " \u001b[32m'fingerprint'\u001b[0m: \u001b[32m'14f938cb78d554e52039879fddb1f1a4'\u001b[0m,\n", " \u001b[32m'details'\u001b[0m: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n", "\u001b[1m}\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import json\n", "from rich import print as rprint\n", "\n", "res = s3_client.get_object(Bucket=s3_bucket, Key=\"manifest-summary.json\")\n", "content = res[\"Body\"].read().decode(\"utf-8\")\n", "print(\"content of manifest summary file:\")\n", "rprint(json.loads(content))" ] }, { "cell_type": "code", "execution_count": 24, "id": "48f51edf-6760-4b11-949b-a6f45e909cc9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "content of manifest data file:\n" ] }, { "data": { "text/html": [ "
\n", "shape: (10, 4)
urietagsizen_record
strstri64i64
"s3://my-bucket/data/1.parquet""..."9673507732709
"s3://my-bucket/data/2.parquet""..."141633791297214
"s3://my-bucket/data/3.parquet""..."10807337832962
"s3://my-bucket/data/4.parquet""..."13934496527477
"s3://my-bucket/data/5.parquet""..."13297418826390
"s3://my-bucket/data/6.parquet""..."10755103779863
"s3://my-bucket/data/7.parquet""..."51982481460156
"s3://my-bucket/data/8.parquet""..."114954021302776
"s3://my-bucket/data/9.parquet""..."12036096890199
"s3://my-bucket/data/10.parquet""..."65631061386662
" ], "text/plain": [ "shape: (10, 4)\n", "┌────────────────────────────────┬──────┬──────────┬──────────┐\n", "│ uri ┆ etag ┆ size ┆ n_record │\n", "│ --- ┆ --- ┆ --- ┆ --- │\n", "│ str ┆ str ┆ i64 ┆ i64 │\n", "╞════════════════════════════════╪══════╪══════════╪══════════╡\n", "│ s3://my-bucket/data/1.parquet ┆ ... ┆ 9673507 ┆ 732709 │\n", "│ s3://my-bucket/data/2.parquet ┆ ... ┆ 14163379 ┆ 1297214 │\n", "│ s3://my-bucket/data/3.parquet ┆ ... ┆ 10807337 ┆ 832962 │\n", "│ s3://my-bucket/data/4.parquet ┆ ... ┆ 13934496 ┆ 527477 │\n", "│ s3://my-bucket/data/5.parquet ┆ ... ┆ 13297418 ┆ 826390 │\n", "│ s3://my-bucket/data/6.parquet ┆ ... ┆ 10755103 ┆ 779863 │\n", "│ s3://my-bucket/data/7.parquet ┆ ... ┆ 5198248 ┆ 1460156 │\n", "│ s3://my-bucket/data/8.parquet ┆ ... ┆ 11495402 ┆ 1302776 │\n", "│ s3://my-bucket/data/9.parquet ┆ ... ┆ 12036096 ┆ 890199 │\n", "│ s3://my-bucket/data/10.parquet ┆ ... ┆ 6563106 ┆ 1386662 │\n", "└────────────────────────────────┴──────┴──────────┴──────────┘" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import polars as pl\n", "\n", "res = s3_client.get_object(Bucket=s3_bucket, Key=\"manifest.parquet\")\n", "df = pl.read_parquet(res[\"Body\"].read())\n", "print(\"content of manifest data file:\")\n", "df" ] }, { "cell_type": "markdown", "id": "5b62c1ec-01c2-4dc9-af08-cf80a210657a", "metadata": {}, "source": [ "## 5. Read Manifest from S3\n", "\n", "When reading a manifest, you only need to provide the **summary file URI**. The library automatically:\n", "\n", "1. Reads the summary file first to get aggregate statistics\n", "2. Uses the reference to locate and read the detailed data file\n", "3. Reconstructs the complete manifest object\n", "\n", "This design enables efficient access patterns for different use cases." ] }, { "cell_type": "code", "execution_count": 25, "id": "fd3065e1-49ab-42ed-b91e-1f788edacee4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "📖 Read manifest from S3:\n", " 📊 Files: 10\n", " 📏 Total size: 102.92 MB\n", " 📦 Total records: 10,036,408\n" ] } ], "source": [ "manifest_file = s3manifesto.ManifestFile.read(\n", " uri_summary=uri_summary,\n", " s3_client=s3_client,\n", ")\n", "\n", "print(f\"📖 Read manifest from S3:\")\n", "print(f\" 📊 Files: {manifest_file.n_data_file}\")\n", "print(f\" 📏 Total size: {manifest_file.size_for_human}\")\n", "print(f\" 📦 Total records: {manifest_file.n_record:,}\")" ] }, { "cell_type": "markdown", "id": "ecfc3b27-92b6-4ba2-8025-945c6616684d", "metadata": {}, "source": [ "## 6. Intelligent File Partitioning by Size\n", "\n", "One of the most powerful features of s3manifesto is **intelligent file partitioning**. This uses the Best Fit Decreasing (BFD) algorithm to group files into balanced batches perfect for parallel processing.\n", "\n", "### Why Partitioning Matters\n", "\n", "- **Consistent Resource Usage**: Each group has similar total size/records\n", "- **Optimal Parallel Processing**: Balanced workloads across workers\n", "- **Memory Planning**: Predictable memory requirements per batch\n", "- **Divide-and-Conquer**: Ready for distributed processing frameworks\n", "\n", "Let's partition our files by size with a target of 30MB per group:" ] }, { "cell_type": "code", "execution_count": 26, "id": "f0982fb3-acf9-49b8-aec8-5ef3b4f707a4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "🔄 Partitioned 10 files into 4 groups\n", "🎯 Target size per group: 30MB\n", "\n", "📦 Group 1: 2 files, 26.80 MB\n", " Files: ['s3://my-bucket/data/2.parquet', 's3://my-bucket/data/4.parquet']\n", "\n", "📦 Group 2: 2 files, 24.16 MB\n", " Files: ['s3://my-bucket/data/5.parquet', 's3://my-bucket/data/9.parquet']\n", "\n", "📦 Group 3: 3 files, 27.53 MB\n", " Files: ['s3://my-bucket/data/8.parquet', 's3://my-bucket/data/3.parquet', 's3://my-bucket/data/10.parquet']\n", "\n", "📦 Group 4: 3 files, 24.44 MB\n", " Files: ['s3://my-bucket/data/6.parquet', 's3://my-bucket/data/1.parquet', 's3://my-bucket/data/7.parquet']\n", "\n" ] } ], "source": [ "groups = manifest_file.partition_files_by_size(\n", " target_size=30_000_000, # 30MB target per group\n", ")\n", "\n", "print(f\"🔄 Partitioned {manifest_file.n_data_file} files into {len(groups)} groups\")\n", "print(f\"🎯 Target size per group: 30MB\")\n", "print()\n", "\n", "for ith, group in enumerate(groups, start=1):\n", " file_count = len(group.data_files)\n", " print(f\"📦 Group {ith}: {file_count} files, {group.size_for_human}\")\n", " \n", " # Show first few filenames for reference\n", " uris = [file.uri for file in group.data_files]\n", " if len(uris) <= 3:\n", " print(f\" Files: {uris}\")\n", " else:\n", " print(f\" Files: {uris[:2]} ... and {len(uris)-2} more\")\n", " print()" ] }, { "cell_type": "markdown", "id": "71f3fa50-9ed3-4950-8ce9-a1ece3ca6a42", "metadata": {}, "source": [ "## 7. Intelligent File Partitioning by Record Count\n", "\n", "You can also partition files by record count, which is useful when processing time is more dependent on the number of records than file size.\n", "\n", "This is particularly valuable for:\n", "- **ETL pipelines** where transformation time scales with record count\n", "- **Database operations** where insert/update performance depends on row count\n", "- **Analytics workloads** where computation scales with data points" ] }, { "cell_type": "code", "execution_count": 27, "id": "ec4e09a4-7452-419c-ac8d-e59556c0cb68", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "🔢 Partitioned by record count: 7 groups\n", "🎯 Target records per group: 2,000,000\n", "\n", "📊 Group 1: 2 files, 1,987,633 records\n", " Total size: 19.1 MB\n", "\n", "📊 Group 2: 1 files, 1,386,662 records\n", " Total size: 6.6 MB\n", "\n", "📊 Group 3: 1 files, 1,302,776 records\n", " Total size: 11.5 MB\n", "\n", "📊 Group 4: 1 files, 1,297,214 records\n", " Total size: 14.2 MB\n", "\n", "📊 Group 5: 2 files, 1,723,161 records\n", " Total size: 22.8 MB\n", "\n", "📊 Group 6: 2 files, 1,606,253 records\n", " Total size: 24.1 MB\n", "\n", "📊 Group 7: 1 files, 732,709 records\n", " Total size: 9.7 MB\n", "\n" ] } ], "source": [ "groups_by_records = manifest_file.partition_files_by_n_record(\n", " target_n_record=2_000_000, # 2M records target per group\n", ")\n", "\n", "print(f\"🔢 Partitioned by record count: {len(groups_by_records)} groups\")\n", "print(f\"🎯 Target records per group: 2,000,000\")\n", "print()\n", "\n", "for ith, group in enumerate(groups_by_records, start=1):\n", " file_count = len(group.data_files)\n", " # Note: group.value represents record count when partitioned by records\n", " print(f\"📊 Group {ith}: {file_count} files, {group.value:,} records\")\n", " \n", " # Calculate total size for this group\n", " total_size = sum(file.size for file in group.data_files)\n", " size_mb = total_size / 1_000_000\n", " print(f\" Total size: {size_mb:.1f} MB\")\n", " print()" ] }, { "cell_type": "markdown", "id": "21799c5d-022a-4920-8bfa-bd1cc7dcb77f", "metadata": {}, "source": [ "## 8. Summary: Key Benefits\n", "\n", "🎯 **Efficient Metadata Management**\n", "- Consolidate scattered file metadata into manageable collections\n", "- Two-file system (JSON summary + Parquet data) for optimal access patterns\n", "- Automatic calculation of aggregate statistics\n", "\n", "⚖️ **Intelligent Partitioning**\n", "- Best Fit Decreasing algorithm ensures balanced groups\n", "- Never exceeds target size/records (except for naturally oversized files) \n", "- Perfect for divide-and-conquer parallel processing\n", "\n", "🚀 **Production Ready**\n", "- Handles millions of files efficiently\n", "- Optimized for AWS S3 storage\n", "- Built for large-scale ETL and data processing workflows\n", "\n", "**Happy data processing**! 🎉" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.10" } }, "nbformat": 4, "nbformat_minor": 5 }