User activities data in a static web application hosted on Azure Kubernetes Services (AKS) needed fast and persistent database reads/writes.

Original solution in 2023 used opensource Redis v6.x with RedisGears for write-behind to a Snowflake database. Though the configuration worked, AKS security constraints made implementation complicated, fragile, and difficult to maintain. After Redis dropped its opensource license in 2024, security vulnerabilities could no longer be kept up to date. Support for RedisGears dwindled, and by mid-2025, a Snowflake table autoincrement issue caused the write-behind process to fail entirely.

When opensource Redis returned in 2025, I saw an opportunity to modernize, streamline, and simplify all parts of the solution using Redis OSS v8.x. My approach used a Stream rather than String data type, made write-behind configurable, and automatically restored data from Snowflake to Redis after a persistent volume failure.

Preserved user data during migration by writing utilities for populating Snowflake tables from Redis dump files. Minimized down-time by populating the missing historical data to the database in advance of the migration. Uploaded 10k records from Redis to Snowflake under 6 minutes with code that leveraged temporary tables and followed bulk loading practices.

Wrote tests and concise README for future self and others engineers team members could read in minutes.

Redis Stream to Snowflake

Instantiate Redis OSS cache and asynchronously move user activity from Redis to Snowflake tables from a pod in Kubernetes with two containers:

1. Redis (main - stores data in stream).
2. Consumer (sidecar - reads stream and writes rows to Snowflake).

Plain Flow

1. UI or API sends a JSON batch with XADD to userdata-stream.
2. Redis stores each batch as a stream entry.
3. Consumer reads entries, builds MERGE / UPDATE SQL, writes to Snowflake.
4. Freshness window skips stale unless allowed.
5. Optional restore seeds Redis from Snowflake when empty.

Terminology

Stream Shapes

Create User:

{"user_email":"user.name@example.com"}

Track Visited / Saved batch:

{
  "key": "userdata-events:user.name-abc123/visited",
  "records": [
    {
      "id": "user.name-abc123",
      "page_name": "search",
      "query_row_count": "25",
      "track_id": "item-01",
      "track_title": "Report 01",
      "track_type": "REPORT",
      "track_url": "/search?q=item-01",
      "lastplus": "2025-11-01 10:20:30",
      "action": "v"
    }
  ]
}

Visited items use key segment visited and action v. Saved uses key segment /saved and action s. Delete uses action d and the database UPDATE sets saved=0.

If track_id is blank and HASH_NULL_TRACK_ID=1, consumer builds a stable auto_<md5prefix> from title|url|id. If disabled it leaves an empty string.

Key Rules

Environment Variables

Restore Logic Summary

Runs once when:

• RESTORE_ON_EMPTY=1
• Stream length = 0 (unless RESTORE_FORCE=1)
• Marker key absent (unless force)

Fetches recent rows, grouped by user+kind, and emits batches respecting KEY_APPEND_PAGE for visited.

Freshness

Records older than NEWER_RECORD_TIME_LIMIT_SECONDS skipped unless:

• Saved and PROCESS_STALE_SAVED=1
• First stale in batch with ALWAYS_PROCESS_STALE_FIRST=1
• SKIP_FRESHNESS=1 disables skip

Track ID

Empty track_id stored as empty string. If HASH_NULL_TRACK_ID=1, computed auto_<12 hex> prefix ensures stable grouping.

Quick Dry Run

python -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
export DRY_RUN=1 REDIS_STREAM=userdata-stream HASH_NULL_TRACK_ID=1
python app/stream_consumer.py &
redis-cli XADD userdata-stream * value '{"user_email":"alice@example.com"}'
redis-cli XADD userdata-stream * value '{"key":"userdata-events:alice-1/visited","records":[{"id":"alice-1","page_name":"search","query_row_count":"5","track_id":"","track_title":"Sample","track_type":"QUERY","track_url":"/search?q=Sample","lastplus":"2025-11-03 12:00:00","action":"v"}]}'

Look for RECORD_EXEC and optionally auto_ hash lines.

Unit Tests

pytest -q

Tests cover:

• JSON field select.
• Parse and repair.
• Username derive.
• Fresh vs stale skip.
• Stale saved override.
• Null track id hashing.
• Import of string keys.
• Failover path.

Common Checks

Troubleshooting Short List

Minimal Deployment Env List

Apply these (example) settings along with Snowflake secrets:

AUTO_QUOTE_LASTPLUS=1
DRY_RUN=0
HASH_NULL_TRACK_ID=1
PROCESS_STALE_SAVED=1
REDIS_CONSUMER=consumer-1
REDIS_GROUP=snowflake-writer
REDIS_STREAM=userdata-stream
SKIP_FRESHNESS=0
STREAM_FAILOVER_TO_SNOWFLAKE=1

Snowflake Verification

Query sample:

SELECT tracked_id, track_id, saved, visited_count, deleted_dts
FROM <DB>.<SCHEMA>.TRACKED
WHERE tracked_id = 'alice-1'
ORDER BY visited_dts DESC
LIMIT 5;

If no rows: check consumer logs for skip or failover markers.

Safe Manual Test in Pod

kubectl exec <pod> -- redis-cli XADD userdata-stream * value '{"diag":"ping","ts":"2025-11-01 10:22:00"}'
kubectl exec <pod> -- redis-cli XLEN userdata-stream

Clean Exit

Stop consumer with Ctrl+C. Redis keeps data.

Persistence

Keeping data on pod restart after persistent volume loss.

Redis writes data to files in /data inside the container. The Kubernetes Persistent Volume (PV) is mounted at that path.

Files:

appendonly.aof (every write is appended; stream entries survive restarts)

dump.rdb (snapshot saved based on save rules)

redis.conf (minimum settings)
plain appendfilename appendonly.aof appendfsync everysec appendonly yes dbfilename dump.rdb dir /data save 300 10 save 60 10000 save 900 1

• appendfsync everysec: flush AOF once per second (small data loss window if a crash happens).
• appendonly yes: log each write (XADD) to AOF.
• dir /data: write files on the mounted volume.
• save xx xx etc: create a snapshot if at least 1 write in 900s, 10 writes in 300s, or 10000 writes in 60s.

Tests

Basic test:

# Add one test entry
TS=$(date -u +"%Y-%m-%d %H:%M:%S")
redis-cli XADD userdata-stream * value '{"diag":"persist","ts":"'$TS'"}'
redis-cli XLEN userdata-stream
# Check files
ls -lh /data
# Tail last lines of AOF
tail -n 5 /data/appendonly.aof

Restart test (in cluster):

1. Record current stream length: redis-cli XLEN userdata-stream.
2. Delete the pod (deployment will recreate it).
3. Run length again: redis-cli XLEN userdata-stream.
4. It should match or be larger.

If length resets to 0:

• PV not mounted in the redis container.
• Wrong path in redis.conf.
• Permissions blocked (check ownership of /data).
• Different DB index used (UI wrote DB 1, you checked DB 0). Test with redis-cli -n 1 XLEN userdata-stream.

To force a snapshot now (optional):

redis-cli SAVE
ls -lh /data/dump.rdb

For near zero loss set appendfsync always (higher disk cost).

If persistence not needed for some environments (local dev), you can keep defaults and ignore PV.

Restore (Snowflake -> Redis Stream on Empty)

Goal: After a pod restart with an empty /data volume the Redis stream would be blank while Snowflake still has authoritative history. The consumer can seed the stream so downstream tools relying on recent backlog have context.

Trigger: Runs once at startup when ALL are true:

• RESTORE_ON_EMPTY=1 (default 1)
• Stream length is 0
• Redis key marker (default _userdata_stream_restored) absent

Process:

1. Query Snowflake TRACKED joined to USER for rows within last RESTORE_DAYS days (default 7).
2. Apply filters for saved / visited based on flags.
3. Limit total rows (RESTORE_LIMIT_TOTAL).
4. Group by user + kind (visited / saved) and emit batches of size RESTORE_BATCH_SIZE as normal stream messages ({"key":"userdata-events:<email>/<visited|saved>","records":[...]}).
5. Set marker key to avoid re-running.

Environment flags:

Dry run mode (DRY_RUN=1) still performs the query but just logs the would-be XADD actions.

To disable completely set RESTORE_ON_EMPTY=0.

Operational note - restore seeds only the stream. It does NOT rewrite Snowflake (source of truth). If restore runs after a true data-loss event for Redis it simply replays recent history for observability/consumers needing backlog.