Delivery Options

Overview

MarketCheck Data Feeds can be delivered through various platforms and protocols to accommodate different infrastructure setups and security requirements. Delivery options fall into two main categories: MarketCheck-owned infrastructure where clients pull data, and client-owned infrastructure where MarketCheck pushes data.

Delivery Frequency

• Daily Feeds: Most data feeds are delivered daily, typically on or before 8 AM UTC for UK markets and 1 PM UTC for North American markets
• Custom Frequencies: Supporting data feeds can be configured for weekly or monthly delivery based on specific requirements

Infrastructure Categories

MarketCheck-Owned Infrastructure

Clients pull data from MarketCheck-managed locations. MarketCheck provides credentials and handles infrastructure maintenance.

Client responsibilities: Use provided credentials and download files within the retention window - no cleanup required.

Client-Owned Infrastructure

MarketCheck pushes data directly to client-managed locations based on delivery frequency.

Client responsibilities: Provide access credentials and permissions, manage storage costs and retention policies, handle cleanup of old files.

Platform Specifications

Google Cloud Storage (GCS)

Credentials Setup:

• Create a Service Account in your GCP project and provide the service account key JSON file
• OR Grant access to MarketCheck's service account: airflow-auth@marketcheck-gcp.iam.gserviceaccount.com

Required Permissions:

AWS S3

Required Credentials:

• AWS Access Key ID
• AWS Secret Access Key
• Region name
• ACL (optional, e.g., bucket-owner-full-control)

Required Permissions:

Documentation References:

• S3 Object Ownership & ACLs
• S3 Policy Actions

Azure Blob Storage

Required Credentials:

• SAS token (Shared Access Signature) OR RBAC user with Storage Blob Data Contributor role

SAS Token Permissions:

RBAC Alternative:

• Assign Storage Blob Data Contributor role instead of SAS token

Documentation Reference:

• Azure Blob Storage Roles

SFTP

Required Credentials:

Filesystem Permissions:

FTP

Required Credentials:

Required Permissions:

Dropbox (Testing Only)

Setup Process:

1. Create a Dropbox App in your account
2. Navigate to Permissions tab and enable required scopes
3. Submit permission changes
4. Generate Access Token from Settings tab

Required Credentials:

• Access Token (from Dropbox App)
• Target Directory path

Required App Permissions:

• files.content.write - Upload and delete files
• files.metadata.read - Read file metadata
• account_info.read - Validate account access

Google Drive (Testing Only)

Setup Process:

1. Create or configure a Shared Google Drive
2. Create a dedicated folder for MarketCheck uploads
3. Grant Content Manager (or Manager) role to the service account
4. Copy the Folder ID from the Drive URL
5. Provide folder ID and target directory path

Required Permissions:

• drive.file or drive.content permissions via folder sharing
• Upload, read metadata, and delete capabilities within the designated folder

File Delivery Details

File Naming Convention

Files follow a standardized naming pattern with feed prefix and date suffix:

• Standard Format: feedname_YYYYMMDD.csv.gz
• Custom Formats: Available based on specific requirements or client conventions

Delivery Locations

• Files can be delivered to different directory paths within the same system
• Multiple storage accounts or containers supported for organizational needs
• Delivery locations require create permissions for MarketCheck

File Retention and Cleanup

MarketCheck-Owned Infrastructure:

• MarketCheck handles cleanup automatically
• Standard retention period: 3 days
• Older files (if needed) need to be generated again and as such require formal request via sales@marketcheck.com

Client-Owned Infrastructure:

• Client responsible for managing file retention
• Client handles cleanup of old files
• No automatic deletion by MarketCheck

Delivery Notifications

MarketCheck provides two notification methods to confirm successful file delivery, that can be used independently or in combination based on your integration requirements to trigger downstream processes.

Webhook Notifications

Webhooks are HTTP callbacks that MarketCheck sends to your specified endpoint immediately after successful file delivery. You need to implement a webhook endpoint on your server to receive these real-time notifications, which can then trigger your downstream data processing workflows.

Basic Webhook Structure:

{
  "status": "SUCCESS",
  "file_name": "mc_us_used_20250721.csv.gz",
  "file_size": "12345678",
  "directory": "[platform-specific-path]"
}

Platform-Specific Directory Fields:

GCS

• In case of Google Cloud Storage (GCS), an additional bucket field is included in payload
• The bucket field specifies the GCS bucket name where the file is stored and directory is the path within that bucket
• Example GCS Payload:

{
  "status": "SUCCESS",
  "file_name": "mc_us_used_20250721.csv.gz",
  "file_size": "12345678",
  "directory": "/daily",
  "bucket": "marketcheck"
}

Authentication:

• Optional X-API-Key header for webhook endpoint security
• HTTPS recommended for secure transmission

Example Webhook Request:

curl -X POST https://mc-hooks.awesomecars.com/ack \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-secret-key" \
  -d '{
    "status": "SUCCESS",
    "file_name": "mc_us_used_20250721.csv.gz",
    "directory": "s3://marketcheck/daily",
    "file_size": "12345678"
  }'

Metafile Notifications

Optional file-based notifications delivered alongside or instead of webhooks:

Success Notification:

• File: mc_us_used_20250721.success.txt
• Contains: Original file size information

Failure Notification:

• File: mc_us_used_20250721.failure.txt
• Contains: Error details and troubleshooting information

See Also

• File Formats
• MarketCheck FTP
• Getting Support