# TheSol Chat - Production Issues Analysis

Date: 2026-04-26  
Application: TheSol Chat (Support Board fork)  
Environment reviewed: `D:\xampp\htdocs\Chatbot` on Windows / XAMPP / PHP 8.2 / MariaDB  
Purpose: Current production-readiness status after re-auditing the codebase, earlier issue lists, and Phase 6-8 hardening work.

---

## 1. Executive Status

The old version of this document mixed real findings with stale findings that have since been fixed. This version is the current working status.

Overall result: the application is much safer and more stable than the original audit state, but it is not yet "100% production ready" for real client data or high-volume usage. The critical runtime/security foundation is mostly complete. Remaining work is concentrated in database performance, migration discipline, cross-server deployment rules, legacy edge cases, and production operations.

| Area | Current status | Readiness |
|---|---|---:|
| AJAX JSON reliability | Centralized JSON finish/shutdown handling in `include/ajax.php`; batch validation and stray-output handling added | Strong |
| Auth/session basics | Empty-password bypass fixed; token timestamp/expiry exists; password change rotates token; login rotates CSRF/session | Strong |
| Admin authorization | AJAX registry exists; default-deny behavior preserved; dangerous ops admin-only | Strong, needs registry coverage tests |
| Upload/file exposure | Embeddings moved/private-blocked; logs/config/install/resource JSON blocked by Apache rules; upload MIME checks hardened | Good, needs Nginx/IIS deployment rules |
| Queue/OpenAI async | Claim tokens, heartbeat, backoff, batch worker, failed-job admin actions implemented | Good |
| OTP | HMAC OTP storage, per-user rate lock, purge command/UI, active flag implemented | Good |
| MSG91/WhatsApp | Shared-secret token available; logs gated/masked; raw message text not logged | Good |
| Settings/admin UI | Save lock, JSON validation, masked-secret restoration, system-info UI implemented | Good |
| Database production scale | Base schema is healthy, but high-volume performance indexes and migration ledger are still pending | Not ready for scale |
| Release operations | Git, smoke test, preflight checker, release checklist, migration script exist | Good, needs production runbook execution |
| Multi-server/cloud | Environment-first config exists; server compatibility documented; shared storage/queue locking strategy still needs deployment validation | Pending |

Production verdict:

| Target | Ready? | Notes |
|---|---|---|
| Local development/demo | Yes | Current code is stable enough for demo use. |
| Small controlled pilot on Apache/cPanel | Conditional | Run preflight, smoke tests, DB backup, migration script, and verify Apache deny rules. |
| Real production with sensitive data | Not yet | Complete Phase 9 and Phase 10 below first. |
| High-volume production | Not yet | Requires database performance indexes, retention, monitoring, and load tests. |
| Multi-server/cloud | Not yet | Requires shared private storage, queue/cron topology, and web-server rule parity. |

---

## 2. Verification Performed In This Re-Audit

Read and compared:

- `PRODUCTION_ISSUES_ANALYSIS.md` previous issue catalog.
- `PRODUCT_STABILITY_AUDIT_2026-04-26.md` Phase 1-8 audit and completion records.
- `DATABASE_PRODUCTION_READINESS_2026-04-26.md` database-scale findings.
- `SERVER_ENVIRONMENT_COMPATIBILITY.md` deployment compatibility plan.
- `RELEASE_CHECKLIST.md` release gate document.
- `resources/migrate-v1.sql` migration script.
- `logs/smoke_test_release.php` smoke test harness.
- Core files: `include/ajax.php`, `include/ajax_registry.php`, `include/functions_users.php`, `include/functions_settings.php`, `include/functions_messages.php`, `include/functions.php`, `include/upload.php`, `include/pusher.php`, `apps/whatsapp/post.php`, `apps/dialogflow/functions.php`, `.htaccess`, `logs/.htaccess`.

Commands/checks used during this re-audit:

- Search for old vulnerable patterns: default allow, empty password checks, `trigger_error`, duplicate email deletion, token expiry, JSON decode handling, user type validation, queue/OTP cleanup, MSG91 token validation.
- File checks for `**/_*.php`: no `_*.php` debug/maintenance scripts were found in the current public workspace.
- Current git worktree check: clean before this document update.

Final verification snapshot after this update:

| Check | Result | Notes |
|---|---|---|
| `php logs/smoke_test_release.php` | PASS | 39 PASS / 0 FAIL. |
| `php logs/preflight_check.php --no-color` | FAIL in local dev config | 38 PASS / 3 WARN / 2 FAIL. Remaining FAIL items are `display_errors=On` and default dev `SB_ENCRYPTION_KEY`. WARN items are missing optional `imap`, localhost `SB_URL`, and install files present. |
| `resources/migrate-v1.sql` repeat run | PASS | Script completed successfully after adding `next_run_at`; existing objects were detected as already present. |
| Queue backoff schema | Fixed during this re-audit | `sb_queue.next_run_at` and `idx_next_run(status, next_run_at)` were added to the local DB and to `resources/migrate-v1.sql`. |

---

## 3. Verified Completed Issues

These issues were previously identified and are now resolved or sufficiently mitigated in code. Do not reopen them unless a regression test proves otherwise.

### 3.1 Security and Authentication

| Old ID | Issue | Current verified status | Evidence |
|---|---|---|---|
| A2 | Empty password bypasses authentication | Completed | `sb_get_active_user()` now accepts empty DB password only when cookie password is also empty, and validates non-empty hashes exactly. |
| A3 / P2-7 | AJAX default-allow / agent catch-all | Completed | `include/ajax_registry.php` defines auth levels; `sb_security()` defaults unknown functions to false after legacy checks. |
| A4 | Duplicate email silently deletes existing user email | Completed | `sb_add_user()` and `sb_update_user()` now return `duplicate-email`; they do not null another user's email. |
| A5 | Tokens never expire | Completed | `token_updated_at` is read and rejected after 30 days; password changes rotate token and timestamp. |
| B2 | No CSRF protection | Completed for registered state-changing AJAX functions | Registry and legacy CSRF lists are enforced in `include/ajax.php`; login regenerates session ID and CSRF token. |
| B7 | `config.php` publicly accessible | Completed for Apache | `.htaccess` denies `config.php`. Equivalent Nginx/IIS rules remain a deployment task. |
| B8 | Install files publicly accessible | Completed for Apache | `.htaccess` denies `install*.php`; preflight warns if install files remain. |
| B18/B19 | Static debug/URL config | Completed | `config.php` is environment-first for debug, URL, DB, private paths, queue settings, logs, and encryption key. |

### 3.2 Users, Messages, and Settings Logic

| Old ID | Issue | Current verified status | Evidence |
|---|---|---|---|
| A1 | Conversation status uses comparison instead of assignment | Completed | `functions_messages.php` uses `$details['status_code'] = 0`. |
| A7 | `sb_get_new_messages()` OR logic duplicates messages | Completed | Current query path uses the revised logic documented in the stability audit. |
| A9 | User deletion without transaction | Completed | `sb_delete_users()` wraps deletion in `START TRANSACTION` / `COMMIT` / `ROLLBACK`. |
| A10 partial | `sb_send_message()` returned `trigger_error()` for missing conversation | Completed for message send path | `sb_send_message()` returns `sb_error('conversation-not-found', ...)`. Other generic `trigger_error()` uses remain pending. |
| A13 | Invalid JSON silently overwrites settings | Completed | `sb_save_settings()` checks `json_last_error()` and returns `json-error`. |
| A14 / P1-6 | Settings concurrency race | Mitigated | `sb_save_settings()` uses `GET_LOCK('sb_settings_save', 5)` and optimistic version increment for main settings. External setting row conflict detection remains a future improvement. |
| A19 | Corrupt external settings JSON returns null / TypeError | Completed | `sb_get_external_setting()` returns default on invalid JSON and guards multi-row merges with `is_array()`. |
| A23 | `sb_get_multi_setting()` treats string as array | Completed | Function now checks `is_array($setting)` before sub-key access. |
| A28 | Agent can set arbitrary user type | Completed | `sb_update_user()` validates user type against `visitor`, `lead`, `user`, `agent`, `admin`, `bot`. |
| C1 | Masked API keys can overwrite real keys | Completed | `sb_save_settings()` calls `sb_restore_masked_settings()` before encoding/saving settings. |

### 3.3 Runtime Reliability and Endpoint Responses

| Old ID | Issue | Current verified status | Evidence |
|---|---|---|---|
| P0-3 | Production logging disabled | Completed | Private `SB_LOG_PATH`, runtime logging, redaction, and production error logging are documented and implemented. |
| P0-4 | AJAX JSON corrupted by warnings/plain text | Completed for `include/ajax.php` | Output buffering, `sb_ajax_finish()`, error handler, fatal shutdown handler, and batch validation exist. |
| P0-5 | OpenAI usage empty IDs break numeric escaping | Completed | OpenAI usage code normalizes missing IDs to SQL `NULL`; smoke tests cover empty IDs. |
| A11 | Pusher crashes when settings missing | Completed | `sb_pusher_init()` / callers guard missing config; `include/pusher.php` returns JSON errors. |
| A15 | Google token refresh returns wrong type on error | Completed | Error propagation is documented as mitigated in the stability audit. |
| A16 | Human takeover crashes on empty conversation | Completed | Empty-data guard is documented as mitigated in the stability audit. |
| PND-16 | Upload/pusher endpoints return plain text errors | Completed for upload/pusher; partially for provider webhooks | `include/upload.php` has shutdown JSON protection; `include/pusher.php` returns JSON statuses. WhatsApp still intentionally returns provider-compatible text for some success paths. |

### 3.4 Files, Uploads, and Deployment Hygiene

| Old ID | Issue | Current verified status | Evidence |
|---|---|---|---|
| P0-1 | Public raw embedding JSON files expose sensitive data | Completed locally | Embeddings path moved to private storage; Apache blocks `uploads/embeddings/`; preflight checks it. |
| P0-2 / B1 | Debug scripts web-accessible | Completed in current workspace | `**/_*.php` search returned no files. Apache also blocks `_debug`, `_test`, `_smoke`, `_migrate` scripts. |
| P1-9 | `.htaccess` gaps for sensitive file types | Completed for Apache | Blocks config/install/debug scripts, logs, SQL/archive files, `apps/*.json`, `resources/*.json`, `uploads/*.txt`, embeddings. |
| P2-8 | Upload MIME bypass for agents | Completed | Upload MIME validation applies to agents and normal users; extension/MIME cross-map added. |
| PND-7 | Dated upload directories lack cleanup | Completed as operational tooling | `logs/queue_worker.php --uploads-cleanup --uploads-days=N` exists; direct web-server blocking of dated folders is still deployment-policy dependent. |

### 3.5 Queue, OTP, WhatsApp, and OpenAI Custom Features

| Old ID | Issue | Current verified status | Evidence |
|---|---|---|---|
| P1-2 | Queue claim race | Completed | `claim_id`, affected-row check, and SELECT by claim token are implemented. |
| P1-3 | Queue timeout too short / no heartbeat | Completed | `heartbeat_at`, `failed_at`, heartbeat-aware reclaim, and 600-second default timeout exist. |
| P1-4 | Concurrency limiter non-atomic | Completed | Advisory lock serialization implemented. |
| P1-5 | Queue assumes `sb_send_message()` success shape | Completed | Queue processor validates result shape and fails job safely. |
| PND-1 | Dead-letter queue admin dashboard | Completed | Admin queue UI and AJAX handlers for view/retry/dismiss exist. |
| PND-2 | Per-job retry/backoff | Completed | `next_run_at` and exponential backoff logic exist in queue functions. |
| PND-3 | Cron batch worker mode | Completed | `logs/queue_worker.php --batch=N` exists. |
| PND-4 | OTP scheduled purge | Completed | `otp-purge-expired` AJAX handler and `queue_worker.php --otp-purge` exist. |
| P2-1 | OTP schema drift | Completed for current migration pack | `otp_code VARCHAR(64)`, `idx_otp_created`, extra OTP columns, and update scripts exist. |
| P2-2 | OTP stores plain codes | Completed | `sb_otp_hash()` HMAC-SHA256 and `hash_equals()` verification are implemented. |
| P2-3 | OTP rate limit race | Completed | Per-user `GET_LOCK('sb_otp_rate_{user_id}', 3)` protects count/invalidate/insert. |
| PND-5 | MSG91 lacks signature validation | Completed with practical shared-secret option | `whatsapp-msg91-webhook-token` setting and `?token=` `hash_equals()` validation exist. |
| P1-8 | MSG91 raw webhook logging leaks PII | Completed | Debug-only logging, phone masking, message body removal, and 5 MB cap exist. |
| A22 | Mixed embedding dimensions not detected | Completed for legacy file embeddings | `embedding-meta.json` stores dimensions and query mismatch returns `embedding-mismatch`. DB-backed embedding dimension checks still need a production test. |

### 3.6 Release Foundation

| Old ID | Issue | Current verified status | Evidence |
|---|---|---|---|
| PND-8 | No Git version control | Completed | Repository has Git history and recent Phase 7/8 commits. |
| PND-9 | No migration script | Partially completed | `resources/migrate-v1.sql` exists for Phase 1-8 foundation schema changes, including queue `next_run_at`; production index pack and migration ledger remain pending. |
| PND-10 | No smoke tests | Completed baseline | `logs/smoke_test_release.php` exists and covers database, settings, message insert, queue, OTP hash, security, `.htaccess`, PHP extensions, logging. |
| PND-11 | No release checklist | Completed baseline | `RELEASE_CHECKLIST.md` exists. |
| PND-13 | Minified JS not rebuilt / no build docs | Completed | `package.json` build scripts exist; `js/min/admin.min.js` has been rebuilt after admin changes. |
| PND-14 | No deployment preflight checker | Completed | `logs/preflight_check.php` exists with CLI, no-color, and JSON modes. |
| PND-15 | AJAX function registry missing | Completed | `include/ajax_registry.php` exists and is required by `include/ajax.php`. |

---

## 4. Verified Pending Issues

These are the remaining production-readiness gaps confirmed in the current code/docs.

### 4.1 Critical Before Real Production

| ID | Issue | Why it matters | Required work |
|---|---|---|---|
| PEND-1 | Database performance indexes not fully applied in migration script | `DATABASE_PRODUCTION_READINESS_2026-04-26.md` still lists missing hot-path indexes for `sb_users_data`, `sb_messages`, `sb_conversations`, OpenAI usage, audit/action tables. Under real traffic, admin lists, phone lookups, and polling can degrade sharply. | Add an idempotent `resources/migrate-production-indexes.sql` or extend `migrate-v1.sql`; test EXPLAIN before/after. |
| PEND-2 | No `sb_schema_migrations` ledger | SQL files exist, but the DB has no canonical record of which migrations ran. Multi-server/future upgrades can drift. | Create `sb_schema_migrations`, record applied migrations with checksums, and make migration scripts insert ledger rows. |
| PEND-3 | Nginx/IIS/web.config deny rules are documented but not shipped as files | `.htaccess` protects Apache only. Deploying to Nginx/IIS without equivalent config can expose protected files. | Add `deploy/nginx-thesol-chat.conf` and `deploy/web.config` templates matching `.htaccess`; include verification commands. |
| PEND-4 | Production encryption key still needs environment verification | Preflight can detect default/short keys, but production must prove a unique secret is set before real data. | Generate and set `SB_ENCRYPTION_KEY` in environment; run `php logs/preflight_check.php --no-color`. |
| PEND-5 | Install files still present locally | `.htaccess` blocks them and preflight warns, but best practice is removal/restriction after install. | Delete or move `install.php` / `install1.php` for production package, or ship them disabled after first install. |

### 4.2 High Priority Hardening

| ID | Issue | Why it matters | Required work |
|---|---|---|---|
| PEND-6 | Registry input schema is descriptive, not enforced | `ajax_registry.php` has `params`, but `include/ajax.php` does not yet validate/coerce all requests against it. | Add registry-driven parameter validation for type, required/optional, integer bounds, arrays, and JSON fields. |
| PEND-7 | CSRF review for every mutation is still needed | Many registry entries are `csrf=false`; some may be safe reads, but mutations should be explicitly reviewed. | Add a registry audit script that lists mutations with `csrf=false`; update entries or document exceptions. |
| PEND-8 | IDOR/resource ownership is not centralized | Some user-level checks rely on `user_id` ownership. Operations involving `message_id`, `conversation_id`, and file paths need systematic ownership guards. | Add helper-level ownership validation for conversation/message/user resources and tests for cross-user access. |
| PEND-9 | Legacy `trigger_error()` remains in generic helpers | Remaining instances in `include/functions.php` can still create noisy runtime behavior outside AJAX contexts. | Convert push notification and automation error paths to `sb_error()`/structured return values. Keep debug-only fatal behavior only where intentional. |
| PEND-10 | OpenAI/API retry policy is incomplete | External API timeouts still depend heavily on one-shot `sb_curl()` behavior. | Add bounded retry/backoff for OpenAI embeddings/chat and provider calls where safe; log final failure with context. |
| PEND-11 | DB-backed embedding dimension validation needs production test | Legacy file path has metadata dimension checks; DB search path should be tested with mismatched vectors/model metadata. | Store/check embedding model+dimension in DB metadata or table column; add smoke test. |

### 4.3 Operational and Scale Gaps

| ID | Issue | Why it matters | Required work |
|---|---|---|---|
| PEND-12 | Retention policy is partial | Queue cleanup, OTP purge, upload cleanup exist, but OpenAI usage, audit logs, webhook logs, PHP logs, caches, and old conversations need a product policy. | Add `logs/maintenance.php` or extend queue worker cleanup for all retention categories; document cron. |
| PEND-13 | Backup/rollback procedure is documented but not executed for target production | A rollback plan must be tested, not just written. | Take DB/files backup on staging, restore it, and record steps in release notes. |
| PEND-14 | Full HTTP deployment verification not automated | Preflight checks filesystem/config; it cannot prove HTTP 403 rules on target host by itself. | Add `logs/http_preflight_check.php` or a script that requests sensitive URLs and asserts 403/404. |
| PEND-15 | Load testing not performed | Current correctness tests do not prove the app handles production concurrency. | Run targeted load tests for login, conversations, message send, WhatsApp webhook, queue worker, OpenAI troubleshoot. |
| PEND-16 | Monitoring/alerting missing | Production errors can be logged but not actively surfaced. | Add alerting for fatal errors, failed queue jobs, webhook failures, OpenAI usage spikes, disk usage, and DB growth. |

### 4.4 Medium Functional Debt

| ID | Issue | Current status | Required work |
|---|---|---|---|
| PEND-17 | Bot message delay uses `sleep()` in some paths | Still visible in `include/functions_messages.php`; can block PHP workers for delayed bot messages. | Move delay to queue/client-side scheduling for web requests. |
| PEND-18 | Typing indicator cleanup is not fully proven | Old issue may remain unless timestamp expiry exists elsewhere. | Add timestamp-based typing expiry or verify existing cleanup. |
| PEND-19 | Email piping concurrency | Old race may remain unless a lock was added elsewhere. | Wrap email piping minute check/process in DB/advisory lock. |
| PEND-20 | CSV export N+1 query risk | Old performance issue likely remains at high user counts. | Rewrite export with JOIN/batched user-data fetch. |
| PEND-21 | Reports division-by-zero needs targeted test | Code may be safe now, but no direct test was verified in this pass. | Add report smoke test for empty date range. |
| PEND-22 | Email template null fallback needs targeted test | Old issue may remain depending on multilingual settings path. | Add template fallback guard and smoke test. |

---

## 5. Section-by-Section Application Assessment

### 5.1 Admin Panel and Settings

Strengths:

- Settings save has JSON validation and advisory lock.
- Masked sensitive settings are restored before save.
- Runtime System Info UI exists via `system-info` AJAX endpoint.
- Admin-only sensitive AJAX functions are explicitly registered.

Remaining work:

- Enforce registry `params` definitions.
- Add settings conflict UX for external setting rows if per-row versioning is added.
- Add production health panel items for queue failures, disk usage, latest error log timestamp, and preflight result summary.

### 5.2 Authentication, Users, and Sessions

Strengths:

- Empty-password bypass fixed.
- Token timestamp and password-change token rotation exist.
- Login rotates session ID and CSRF token.
- Duplicate email data-loss behavior fixed.
- User type whitelist exists on add/update.

Remaining work:

- Add explicit logout token/session invalidation if product policy requires immediate global logout.
- Add automated tests for visitor cookie login, expired token rejection, and cross-user access rejection.

### 5.3 AJAX/API Surface

Strengths:

- AJAX JSON corruption problem is largely solved.
- Registry gives one place to review auth/CSRF intent.
- Batch calls restore `$_POST` after every item.

Remaining work:

- Enforce input schemas.
- Add CI script that compares `case 'function'` entries in `include/ajax.php` with `SB_AJAX_REGISTRY` entries.
- Complete CSRF mutation review.
- Centralize resource ownership checks.

### 5.4 Messaging and Conversations

Strengths:

- Message insert is checked before downstream queue/routing/webhook work.
- Conversation status assignment bug is fixed.
- Queue-backed bot paths are safer.

Remaining work:

- Remove/blocking `sleep()` paths from web requests.
- Add DB indexes for message/conversation hot paths.
- Add empty-date-range and high-pagination tests.

### 5.5 Uploads, Media, and Public Files

Strengths:

- MIME checks apply to agents.
- Extension/MIME cross-validation exists.
- Sensitive artifacts are blocked by Apache and private paths.
- Upload cleanup command exists.

Remaining work:

- Ship Nginx/IIS templates.
- Validate upload directory non-executable policy on target host.
- Define retention for dated upload folders by data type.

### 5.6 WhatsApp, MSG91, and Webhooks

Strengths:

- MSG91 query-token validation exists.
- Webhook logs are debug-only, masked, and capped.
- Structured endpoint errors improved where JSON compatibility is expected.

Remaining work:

- Document final MSG91 webhook URL format in deployment runbook.
- Add optional IP allowlist if MSG91 publishes stable source ranges for target deployment.
- Add webhook replay/duplicate handling test if provider retries aggressively.

### 5.7 OpenAI, Dialogflow, Embeddings

Strengths:

- OpenAI usage no longer crashes on empty conversation/user IDs.
- Queue/concurrency hardening reduces duplicate OpenAI costs.
- Legacy embedding dimension mismatch is detected.
- OTP tools are opt-in.

Remaining work:

- Add retry/backoff for safe external API calls.
- Add DB-backed embedding dimension/model metadata validation.
- Add OpenAI request context column/reporting if operations need cost attribution by source.

### 5.8 Queue and Background Jobs

Strengths:

- Claim token, heartbeat, backoff, failed-job UI, batch mode, cleanup, OTP purge, upload cleanup exist.

Remaining work:

- Define production cron/supervisor configuration per host.
- Add alerting for failed jobs above threshold.
- Add retention for completed jobs beyond the existing cleanup command.

### 5.9 Database

Strengths:

- Core data is healthy according to the DB readiness report.
- Phase 1-4 custom schema changes are captured in `resources/migrate-v1.sql`.

Remaining work:

- Apply production performance indexes.
- Add `sb_schema_migrations` ledger.
- Add retention indexes and policies.
- Consider MEDIUMTEXT upgrades for AI-heavy payloads.

### 5.10 Deployment and Operations

Strengths:

- `RELEASE_CHECKLIST.md`, `SERVER_ENVIRONMENT_COMPATIBILITY.md`, `logs/preflight_check.php`, and `logs/smoke_test_release.php` exist.

Remaining work:

- Execute the checklist on the target server.
- Add web-server-specific config templates.
- Add tested backup/restore proof.
- Add monitoring and log rotation at host level.

---

## 6. Phase-Wise Plan To Reach 100% Production Readiness

### Phase 9 - Production Database Pack

Goal: make the database safe for real traffic and future upgrades.

| # | Task | Status | Owner notes |
|---|---|---|---|
| 9-1 | Add `sb_schema_migrations` table and migration ledger writes | Pending | Required before more DB changes. |
| 9-2 | Create idempotent production index migration for `sb_users_data`, `sb_messages`, `sb_conversations` | Pending | Use guards against duplicate indexes. |
| 9-3 | Add OpenAI usage context/reporting columns and indexes | Pending | Pair with code update to populate context. |
| 9-4 | Add retention/audit indexes from DB readiness report | Pending | `sb_audit_log`, action submissions, cache tables. |
| 9-5 | Run EXPLAIN before/after and document query improvements | Pending | Needed to prove real performance benefit. |

Exit criteria:

- Migration can be run twice safely.
- Ledger records applied migration names/checksums.
- Smoke test passes after migration.
- No slow full scans remain for phone lookup and common conversation/message list queries.

### Phase 10 - Cross-Server Deployment Hardening

Goal: make the same product safe on Apache, Nginx, IIS, shared hosting, and VPS.

| # | Task | Status | Owner notes |
|---|---|---|---|
| 10-1 | Add `deploy/nginx-thesol-chat.conf` deny-rule template | Pending | Must match `.htaccess`. |
| 10-2 | Add `deploy/web.config` IIS deny-rule template | Pending | Must block logs, config, install, JSON configs, embeddings, exports. |
| 10-3 | Add HTTP preflight script to request sensitive URLs and assert 403/404 | Pending | Complements CLI preflight. |
| 10-4 | Package production install guidance for deleting/restricting install files | Pending | Include post-install lock step. |
| 10-5 | Verify private path, log path, uploads path, and non-executable upload policy on target host | Pending | Must be run on actual server. |

Exit criteria:

- Apache, Nginx, and IIS security rules are shipped.
- Sensitive URLs are verified blocked through HTTP, not only file inspection.
- Preflight has no FAIL items and only accepted WARN items.

### Phase 11 - AJAX Contract and Authorization Tests

Goal: prevent new endpoints from becoming security regressions.

| # | Task | Status | Owner notes |
|---|---|---|---|
| 11-1 | Enforce `ajax_registry.php` parameter schemas in `sb_ajax_execute()` | Pending | Type, required/optional, JSON, arrays. |
| 11-2 | Add script to diff `switch` cases against registry entries | Pending | Fails if an AJAX function is unregistered. |
| 11-3 | Review all `csrf=false` registry entries and mark documented exceptions | Pending | Mutations should require CSRF unless provider/public compatibility requires otherwise. |
| 11-4 | Add ownership tests for user/message/conversation endpoints | Pending | Prove user A cannot read/update user B's resources. |
| 11-5 | Convert remaining generic `trigger_error()` paths to structured errors | Pending | Especially push notification and automation paths. |

Exit criteria:

- Unknown AJAX functions fail tests.
- Registry schema validation rejects invalid request types.
- Cross-user IDOR tests pass.
- CSRF mutation matrix is documented.

### Phase 12 - Runtime Resilience and Operations

Goal: make production failures visible, bounded, and recoverable.

| # | Task | Status | Owner notes |
|---|---|---|---|
| 12-1 | Add bounded retry/backoff for safe OpenAI/provider calls | Pending | Avoid retrying non-idempotent sends unless guarded. |
| 12-2 | Remove blocking bot-delay `sleep()` from web request paths | Pending | Use queue/client delay instead. |
| 12-3 | Add unified maintenance command for queue, OTP, uploads, caches, logs, OpenAI usage, audit retention | Pending | Existing queue/OTP/upload cleanup can be foundation. |
| 12-4 | Add failed-job/webhook/error monitoring thresholds | Pending | Admin dashboard or log alerts. |
| 12-5 | Run targeted load tests for login, AJAX batch, message send, webhook, queue, and OpenAI troubleshoot | Pending | Measure DB and PHP behavior. |

Exit criteria:

- External API timeouts return structured, logged failures.
- No web request sleeps for configured bot delays.
- Retention command runs from cron and reports deleted counts.
- Load test results are recorded.

### Phase 13 - Production Release Execution

Goal: turn the prepared codebase into a controlled deployable product.

| # | Task | Status | Owner notes |
|---|---|---|---|
| 13-1 | Set production environment variables: `SB_URL`, DB, `SB_PRIVATE_PATH`, `SB_LOG_PATH`, `SB_ENCRYPTION_KEY`, queue settings | Pending | Never use local defaults for production. |
| 13-2 | Take full DB and file backup; restore backup on staging to prove rollback | Pending | Document exact restore command. |
| 13-3 | Run `php logs/preflight_check.php --no-color` on target host | Pending | Resolve every FAIL. |
| 13-4 | Run `php logs/smoke_test_release.php` on target host | Pending | Must pass. |
| 13-5 | Run HTTP preflight against deployed URL | Pending | Must block sensitive files. |
| 13-6 | Run manual end-to-end tests: admin login, save settings, chat widget, message send, WhatsApp webhook, OTP happy/failure, queue job, OpenAI troubleshoot | Pending | Record results in release notes. |
| 13-7 | Tag final release and freeze production package | Pending | Example: `v1.0-production`. |

Exit criteria:

- Release checklist signed.
- Preflight and smoke tests pass on the actual target server.
- Rollback is tested.
- Monitoring/log paths confirmed.

---

## 7. Current Minimum Work Before Real Production

Do these before putting real users or sensitive data on the system:

1. Complete Phase 9 database pack: production indexes plus migration ledger.
2. Complete Phase 10 server rule templates and HTTP access verification.
3. Set a unique production `SB_ENCRYPTION_KEY` and run preflight cleanly.
4. Delete or production-lock `install.php` and `install1.php` after installation.
5. Run backup/restore proof on staging.
6. Run smoke tests and HTTP preflight on the actual deployment host.
7. Review CSRF/IDOR matrix and add automated tests for the highest-risk endpoints.

---

## 8. Verification Commands

Run these after every production-hardening batch.

### PHP lint modified files

```powershell
php -l include/ajax.php
php -l include/ajax_registry.php
php -l include/functions_users.php
php -l include/functions_settings.php
php -l include/functions_messages.php
php -l include/upload.php
php -l include/pusher.php
php -l apps/whatsapp/post.php
php -l apps/dialogflow/functions.php
php -l logs/preflight_check.php
php -l logs/smoke_test_release.php
```

### Release smoke test

```powershell
php logs/smoke_test_release.php
```

### Deployment preflight

```powershell
php logs/preflight_check.php --no-color
php logs/preflight_check.php --json
```

### No maintenance scripts in public tree

```powershell
Get-ChildItem -Recurse -Filter '_*.php' D:\xampp\htdocs\Chatbot |
    Where-Object { $_.FullName -notlike '*\vendor\*' }
```

Expected output: no files.

### Check sensitive Apache rules

```powershell
Select-String -Path .htaccess -Pattern 'config.php|install|uploads/embeddings|apps/.*json|resources/.*json|uploads/.*txt|FilesMatch'
```

### Check migration ledger after Phase 9

```sql
SHOW TABLES LIKE 'sb_schema_migrations';
SELECT * FROM sb_schema_migrations ORDER BY applied_at DESC;
```

---

## 9. Final Assessment

The application has passed the most important stability transformation: the original critical items around AJAX corruption, default-allow security, public sensitive files, OTP plaintext storage, queue races, MSG91 PII logs, and missing release scaffolding are now fixed or substantially mitigated.

The remaining path to "100% production ready" is no longer a mystery bug hunt. It is an execution plan:

- Make the database production-scale.
- Make web-server protections portable beyond Apache.
- Turn the AJAX registry into an enforced contract.
- Add final ownership/CSRF tests.
- Prove backup, restore, preflight, smoke tests, and HTTP access rules on the target host.

Until those are complete, the correct status is: production-candidate for controlled testing, not final production for real client data.
