# NGINX Custom-to-Combined Log Converter

This kit converts the observed custom RockyWeb/NGINX `access_log` format into standard NGINX `combined` format.

## Observed formats

### Input: custom_main format

```text
"$remote_addr" - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$custom_field_2" "$http_user_agent" "$custom_field_4"
```

Example:

```text
"20.224.136.216" - - [12/Jul/2026:00:01:34 -0600] "GET /path/file.xml HTTP/1.1" 200 11943 "-" "-" "python-requests/2.34.2" "-"
```

### Output: NGINX combined format

```text
$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"
```

Example:

```text
20.224.136.216 - - [12/Jul/2026:00:01:34 -0600] "GET /path/file.xml HTTP/1.1" 200 11943 "-" "python-requests/2.34.2"
```

## Field mapping

| Output combined field | Source custom field |
|---|---|
| `$remote_addr` | first quoted field, with quotes removed |
| literal `-` / ident | second token after source IP |
| `$remote_user` | third token after source IP |
| `$time_local` | bracketed timestamp |
| `$request` | quoted request field |
| `$status` | status token |
| `$body_bytes_sent` | byte-count token |
| `$http_referer` | first quoted field after bytes |
| `$http_user_agent` | third quoted field after bytes |

The second and fourth quoted fields after bytes are retained only in the input parser and are intentionally dropped because standard combined format has no slots for them.

## Convert a single file

```bash
python convert_custom_main_to_combined.py \
  --input /path/to/custom_main.txt \
  --output /path/to/combined_logs \
  --workers 1 \
  --overwrite
```

Output path:

```text
/path/to/combined_logs/custom_main.txt.combined
```

## Convert a directory of logs

```bash
python convert_custom_main_to_combined.py \
  --input /data/custom_logs \
  --output /data/combined_logs \
  --recursive \
  --pattern 'access_log*' \
  --workers 24 \
  --overwrite
```

## Convert compressed logs and write compressed output

```bash
python convert_custom_main_to_combined.py \
  --input /data/custom_logs \
  --output /data/combined_logs \
  --recursive \
  --pattern 'access_log*' \
  --workers 24 \
  --gzip-output \
  --overwrite
```

For input `access_log.07-12-26.gz`, this writes:

```text
access_log.07-12-26.combined.gz
```

## Validate output logs

```bash
python convert_custom_main_to_combined.py \
  --input /data/combined_logs \
  --recursive \
  --pattern '*.combined*' \
  --validate-only \
  --workers 24
```

Validation exits with status `0` when all lines match either the custom parser or the combined parser. It exits with non-zero status if any lines are unmatched.

## Diagnostics

The converter writes:

```text
conversion_manifest.csv
```

The manifest contains one row per input file with total line count, converted custom line count, already-combined pass-through count, unmatched line count, byte counts, output path, and status.

When unmatched lines are found, the converter writes a sibling `.bad` file containing:

```text
line_number<TAB>original_line
```

## Notes

- The converter streams each file line-by-line and does not load whole logs into memory.
- The converter does not deduplicate. Every physical input line is converted or marked bad.
- Already-combined lines are passed through by default, which makes the converter safe for mixed directories. Use `--no-pass-through-combined` to reject already-combined lines.