CodeSolutionsHub

Redis is a popular in-memory key-value data store often used with Apache Airflow for tasks like caching and message brokering. While using Redis in Docker Compose is straightforward, connect to local redis from docker container can pose unique challenges, especially for Linux users. In this article, we’ll walk through a practical solution to seamlessly integrate your local Redis instance with your Airflow Docker container.

The Challenge

When running Airflow in a Docker container, users often need to connect it to a Redis instance running on their local machine. This setup can be tricky for several reasons:

1. Container Isolation: By default, Docker containers run in an isolated network and cannot directly access the host’s localhost.
2. Linux-Specific Issues: While Docker Desktop on macOS and Windows provides the host.docker.internal alias for accessing the host, this feature isn’t natively available on Linux.
3. Networking Configurations: Incorrect configurations in Redis or Docker can lead to connection issues.

The Solution

We’ll address these issues by using Docker’s host network mode. This allows a container to share the host’s network namespace, enabling direct access to services running on the host machine’s localhost.

Step 1: Update Redis Configuration

First, ensure your local Redis instance is configured to accept connections:

1. Open your Redis configuration file (e.g., /etc/redis/redis.conf).
2. Update the bind directive to allow connections from all interfaces:
   bind 0.0.0.0
3. Restart Redis to apply the changes:
   sudo systemctl restart redis
4. Verify Redis is running and accessible:
   redis-cli -h 127.0.0.1 ping
   You should see a response: PONG.

Step 2: Modify Airflow’s Docker-Compose File

Update your docker-compose.yml to enable the host network mode for the Airflow container:

version: "3"
services:
  airflow-webserver:
    image: apache/airflow:2.6.2
    container_name: airflow-webserver
    environment:
      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://postgres:password@postgres/airflow
      - AIRFLOW__CORE__EXECUTOR=LocalExecutor
      - AIRFLOW_CONN_REDIS=redis://localhost:6379
    ports:
      - "8080:8080"
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./plugins:/opt/airflow/plugins
    depends_on:
      - postgres
    network_mode: host  # Use host network mode
    restart: always

  postgres:
    image: postgres:13
    container_name: postgres
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - airflow_network

volumes:
  postgres_data:

networks:
  airflow_network:
    driver: bridge

Key Changes:

• Added network_mode: host to the airflow-webserver service.
• Configured AIRFLOW_CONN_REDIS to point to redis://localhost:6379, enabling Airflow to connect to the local Redis instance.

Step 3: Restart Airflow

After updating the docker-compose.yml, restart the Airflow container:

docker-compose down
docker-compose up -d

This ensures the changes take effect.

Step 4: Test the Redis Connection

To confirm that Airflow can connect to Redis:

1. Log into the Airflow Web UI (default: http://localhost:8080).
2. Go to Admin > Connections and verify that the Redis connection is properly configured.
3. Test the connection by writing a simple Python script in your Airflow DAG:

from redis import Redis

def test_redis_connection():
    redis_host = "localhost"
    redis_port = 6379

    try:
        client = Redis(host=redis_host, port=redis_port)
        client.ping()
        print("Connected to Redis!")
    except Exception as e:
        print(f"Failed to connect to Redis: {e}")

If the connection is successful, you’ll see the message: Connected to Redis! in the logs.

Common Pitfalls and Solutions

1. Redis Connection Refused

• Cause: Redis may not be listening on all interfaces.
• Fix: Double-check the bind configuration in
  /etc/redis/redis.conf

2. host.docker.internal Not Working on Linux

• Cause: Linux doesn’t natively support
  host.docker.internal.
• Fix: Use the host network mode as outlined above or add --add-host in your Docker Compose file:
  extra_hosts: - "host.docker.internal:host-gateway"

3. Firewall Blocking Redis

• Cause: Firewall rules may prevent external connections to Redis.
• Fix: Allow connections on port 6379:sudo ufw allow 6379

Why This Solution Works

Using the host network mode eliminates the complexities of container-to-host networking by allowing the container to directly access the host’s localhost. This approach is simple, effective, and works seamlessly on Linux systems where host.docker.internal is not available.

Conclusion

Integrating a locally running Redis instance with a Dockerized Airflow setup doesn’t have to be a headache. By using Docker’s host network mode, you can bridge the gap between your host machine and containerized applications effortlessly. Follow the steps in this guide, and you’ll have a fully functional connection between Airflow and Redis in no time.

Have questions or faced a unique issue? Let us know in the comments below!

Checkout more common docker related issues here