Purpose
This guide explains how to configure the Custom S3 (S3 Compatible) poller in Logsign USO to pull log objects from any S3-compatible object storage, not just Amazon S3. Use this integration for storages such as MinIO, Ceph, Backblaze B2, Cloudflare R2 (generic access, not the dedicated Cloudflare R2 integration), NetApp StorageGRID, Wasabi, or any other product that exposes an S3-compatible API. If you specifically need Amazon AWS S3, use the separate "AWS S3 Integration via API" article instead, that integration is tuned for AWS's own IAM/credential model.
Logsign lists new objects in the configured bucket on each poll and ingests them as they appear. The poller auto-detects common compression formats and, for storages that deliver one JSON array per object instead of newline-delimited records, expands the array into individual events.
Prerequisites
- An access key and secret key on the target storage with permission to list and read objects in the bucket you want to collect from.
- The storage's S3-compatible API endpoint URL (for example
https://minio.example.com:9000). - Network connectivity from the Logsign server to that endpoint.
- The bucket name. If your credentials are scoped to a single bucket and cannot list all buckets on the account, have the exact bucket name ready to type in manually (see the Bucket Name field below).
Configure in Logsign
- Go to Settings > Integrations > Data Collection, then Add Device.
- Search for and select Custom S3 (S3 Compatible) as the device type.
- Fill in the fields below, then click Check Connection. A successful test reveals the Bucket Name field (as an autocomplete populated from the buckets your credentials can list) along with the advanced fields (Start After Key, Bucket Prefix).
| Field | Required | Description |
|---|---|---|
| Endpoint URL | Yes | The S3-compatible API endpoint of your storage, including scheme and port if non-standard, for example https://minio.example.com:9000. |
| Access Key | Yes | The access key ID for the storage account. |
| Secret Key | Yes | The secret access key paired with the Access Key. Stored encrypted at rest. |
| Region | No | Defaults to auto. Most S3-compatible storages accept auto or us-east-1; check your storage vendor's own documentation if connections fail with a region-related error. |
| Use path-style addressing | No | Enabled by default. Keep this on for self-hosted/on-prem storages such as MinIO or Ceph. Disable it only for services that require virtual-hosted-style addressing (bucket name as part of the hostname rather than the path). |
| Verify TLS certificate | No | Enabled by default. Disable only if the storage serves a self-signed certificate that Logsign cannot otherwise validate. |
| Bucket Name | Yes | The bucket to collect from. Populated as a searchable list after a successful connection test, drawn from the buckets your credentials can list. If your credentials are bucket-scoped and cannot list buckets, type the exact bucket name manually before running the connection test. |
| Start After Key | No | Advanced field shown after a successful connection test. Seeds the initial position: only objects sorted after this key (lexicographically) are collected on the first poll. Leave empty to start from the beginning of the bucket (subject to Bucket Prefix, if set). Not needed on ongoing polls; Logsign tracks the last collected object key automatically after the first run. |
| Bucket Prefix | No | Advanced field shown after a successful connection test. Restricts collection to objects whose key starts with this prefix, useful when a bucket is shared across multiple log types or tenants and only a subfolder should be ingested. |
Notes and Limits
- Supported object formats: plain text (newline-delimited records), gzip, and JSON. An object holding a single JSON array with no line breaks is detected and expanded into one event per array element rather than being treated as one oversized record.
- Objects in a compression or container format this connector does not decode (zip, bzip2, zstd, xz, lz4, parquet, avro) are still ingested rather than silently dropped, but a warning is logged naming the detected format so the source can be reviewed.
- A single log line/record is capped at 250KB; content beyond that is not read for that record.
- This is a generic, vendor-agnostic collector: field mapping and event classification depend entirely on the log content your storage produces and are not pre-mapped to any specific vendor's schema the way a dedicated integration (AWS S3, Cloudflare R2) would be. Expect to build or request custom parsing/mapping for the log format your storage writes.
- Object listing uses the storage's own pagination; very large buckets are paged through automatically and do not need to be split manually.
Troubleshooting
| Symptom | Likely cause | What to check |
|---|---|---|
| Check Connection fails | Wrong Endpoint URL, Access Key, or Secret Key; or the storage requires a specific Region value. | Re-verify all four fields against your storage console. Try the region value your storage vendor documents if auto does not work. |
| Check Connection succeeds but the Bucket Name list is empty | The credentials are scoped to a single bucket and cannot call the storage's list-buckets operation. | Type the bucket name manually into the Bucket Name field instead of picking from the list. |
| Connection test fails with a TLS/certificate error | The storage serves a self-signed or internally-issued certificate. | Enable "Verify TLS certificate" only if you trust the network path; disabling it removes certificate validation entirely. |
| No new events despite new objects in the bucket | Bucket Prefix is set and excludes the objects being written, or Start After Key was seeded past the objects you expect to see. | Check the Bucket Prefix value against the actual object keys in the bucket, and confirm Start After Key (if set) sorts before the objects you expect. |
| Some objects are ingested as one oversized or malformed-looking record | The object is in a compression/container format this connector does not decode (see Notes and Limits). | Check the poller log for a warning naming the detected format for that object. |