Trojan Reuse on Port 443

Posted on 2020-05-17 Edited on 2021-02-08 In Experiment Word count in article: 2.4k Reading time ≈ 9 mins.

HTTPS has become the fundamental threshold for internet service access, and the use of port 443 as the default port for HTTPS requests, combined with virtual hosting support, seamlessly complements this.

However, Trojan is somewhat exceptional. Its operational mode necessitates direct interfacing with traffic entry points; otherwise, its protocol cannot be correctly identified by the server. Additionally, for the sake of enhancing service concealment, it’s often configured on port 443. Nevertheless, there’s only one port 443 available. While Trojan offers a “non-standard request” forwarding feature, it’s still relatively new, and channeling all traffic through it presents limitations in terms of stability, performance, flexibility, and, importantly, it doesn’t support TLS forwarding.

In this context, I’ve devised a scheme for sharing port 443 between Trojan and Nginx, with Docker deployment support.

Let’s start by discussing the final architectural setup:

graph LR
A["Client"] -->|HTTPS Request| B("Nginx")
B --> C{"dispatch"}
C -->D["Trojan"]
C -->E["V2Ray"]
C -->F["Web"]
D -->|Invalid Requst| G["Default Page"]
E -->|Invalid Requst| G["Default Page"]

The primary task here is the flow distribution during the dispatch phase. Since it involves HTTPS traffic, some relevant knowledge is necessary to determine the distribution strategy.

Before the popularity of virtual hosting, a service IP would typically have only one TLS service site and one SSL certificate. Consequently, there was only one certificate for incoming requests.

However, with the advent of virtual hosting, a single service IP can host multiple TLS service sites, each with its SSL certificate. This raises the question: how to ascertain which certificate to use for a particular request?

This is where SNI (Server Name Indication) comes into play. SNI requires the client to specify which site it wants to connect to during the initial TLS handshake when there are multiple TLS service sites on a single IP. In practice, this is achieved by adding a server_name field in the Client Hello phase.

TLS SNI

So, SNI serves as the foundation for supporting multiple HTTPS hosts on the same IP:

server {
    listen          443 ssl;
    server_name     www.chengxiaobai.com;
    ssl_certificate www.chengxiaobai.com.crt;
}
 
server {
    listen          443 ssl;
    server_name     nothing.chengxiaobai.com;
    ssl_certificate nothing.chengxiaobai.com.crt;
}

Our traffic distribution strategy is based on this data.

Firstly, it’s essential to clarify that Trojan cannot be proxied by Nginx at the 7th layer. Therefore, it must operate at the traffic entry point, and Nginx can only be placed behind it.

I still have a question: Why can’t we use proxy_pass under the http module to perform HTTPS reverse proxy for Trojan?

I tried it and found that the connection failed. I suspect that Trojan itself performs data verification during the connection establishment process or does not fully implement the HTTP protocol (HTTP CONNECT part)?

The proxy_pass in the http module works at the 7th layer and may require mounting certificates for traffic decryption and recognition, followed by re-encrypting the request. It’s essentially acting as a Man-in-the-Middle (MITM) proxy. Alternatively, the connection failure could be due to using HTTP CONNECT to establish a tunnel. There are many complexities involved, and perhaps examining the source code might shed light on it.

Since we can’t forward at the 7th layer, we’ll resort to forwarding at the 4th layer.

Nginx supports SNI-based 4th layer forwarding, which involves recognizing SNI information and directly forwarding TCP/UDP data streams. This feature is significantly more powerful than 7th layer virtual hosting. It’s provided by the ngx_stream_ssl_preread_module module, but it’s not enabled by default. Configuring it is straightforward, but it’s important to note that this module belongs to stream, not the commonly used http.

Here’s the Nginx configuration:

user  nginx;

pid   /var/run/nginx.pid;

# Keep the rest of the configuration default

# Core configuration for traffic distribution
stream {
    # SNI recognition to map domain names to configurations
    map $ssl_preread_server_name $backend_name {
        web.com.chengxiaobai web;
        vmess.com.chengxiaobai vmess;
        trojan.com.chengxiaobai trojan;
    # Default value when no domain matches
        default web;
    }

    # Configuration details for 'web'
    upstream web {
        server 127.0.0.1:10240;
    }

    # Configuration details for 'trojan'
    upstream trojan {
        server 127.0.0.1:10241;
    }

    # Configuration details for 'vmess'
    upstream vmess {
        server 127.0.0.1:10242;
    }

    # Listen on port 443 and enable ssl_preread
    server {
        listen 443 reuseport;
        listen [::]:443 reuseport;
        proxy_pass  $backend_name;
        ssl_preread on;
    }
}

http {
  # Keep this section unchanged
}

With just a few lines of configuration, we’ve achieved traffic distribution. Finally, adjust the configuration ports for Trojan and V2Ray to match the one above.

For V2Ray, here’s the Nginx configuration, which decrypts TLS encryption information to improve overall link performance and enable site spoofing:

server {
    # Enable HTTP2 support
    listen 10242 ssl http2;
    server_name  vmess.com.chengxiaobai;
  
    gzip on;
    gzip_http_version 1.1;
    gzip_vary on;
    gzip_comp_level 6;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/javascript application/x-javascript text/javascript;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_certificate      /etc/ssl/vmess.com.chengxiaobai.crt;
    ssl_certificate_key  /etc/ssl/private/vmess.com.chengxiaobai.key;
    ssl_session_cache    shared:SSL:1m;
    ssl_session_timeout  5m;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

    # WS protocol forwarding
    location /fTY9Bx7c {
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
        proxy_pass http://127.0.0.1:16881;
    }

    # Forward non-standard requests
    location / {
       proxy_pass http://127.

0.0.1:80;
    }
}

And here’s the V2Ray configuration:

{
  "api": {
    "tag": "api"
  },
  "inbounds": [
    {
      "allocate": {
        "strategy": "always"
      },
      "port": 16881,
      "protocol": "vmess",
      "settings": {
        "clients": [
          {
            "alterId": 64,
            "id": "Unique ID",
            "level": 1
          }
        ]
      },
      "sniffing": {
        "enabled": false
      },
      "streamSettings": {
        "network": "ws",
        "security": "none", // Note that there's no TLS here, as TLS is handled by Nginx
        "wsSettings": {
          "path": "/fTY9Bx7c" // You can make it more complex to avoid detection
        }
      }
    }
  ],
  "log": {
    "access": "/var/log/v2ray/access.log",
    "error": "/var/log/v2ray/error.log",
    "loglevel": "error"
  },
  "outbounds": [
    {
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "AsIs"
      },
      "tag": "direct"
    },
    {
      "protocol": "blackhole",
      "settings": {
        "response": {
          "type": "none"
        }
      },
      "tag": "blocked"
    }
  ],
  "policy": {},
  "reverse": {},
  "routing": {
    "domainStrategy": "IPIfNonMatch",
    "rules": [
      {
        "ip": [
          "geoip:private"
        ],
        "outboundTag": "blocked",
        "type": "field"
      }
    ]
  },
  "stats": {},
  "transport": {}
}

Trojan Configuration:

{
  "run_type": "server",
  "local_addr": "127.0.0.1",
  "local_port": 10241,
  "remote_addr": "127.0.0.1", // Forward to Nginx default page
  "remote_port": 80,
  "password": [
    "password"
  ],
  "log_level": 3,
  "ssl": {
    "cert": "certificate_address.crt",
    "key": "certificate_address.key",
    "key_password": "",
    "cipher": "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384",
    "cipher_tls13": "TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384",
    "prefer_server_cipher": true,
    "alpn": [
      "http/1.1"
    ],
    "alpn_port_override": {
      "h2": 81
    },
    "reuse_session": true,
    "session_ticket": false,
    "session_timeout": 600,
    "plain_http_response": "",
    "curves": "",
    "dhparam": ""
  },
  "tcp": {
    "prefer_ipv4": false,
    "no_delay": true,
    "keep_alive": true,
    "reuse_port": false,
    "fast_open": false,
    "fast_open_qlen": 20
  },
  "mysql": {
    "enabled": false,
    "server_addr": "127.0.0.1",
    "server_port": 3306,
    "database": "trojan",
    "username": "trojan",
    "password": "",
    "cafile": ""
  }
}

For other services, you can use the regular configuration as before. After all, Nginx can forward to Trojan and also reverse-proxy other services. The traffic distribution architecture I’m currently using is as follows:

graph LR
A["Client"] -->|HTTPS Request| B("Nginx")
B -->C{"dispatch"}
C -->|TCP Forward| D["Trojan"]
C -->|TCP Forward|E0["Nginx WS Port"]
C -->|TCP Forward|F0["Nginx Web Port"]
E0-->|WS Protocol|E["V2Ray"]
E0-->|Invalid Request| G0["Nginx Default Port"]
F0-->|HTTP|F["PHP"]
D -->|Invalid Request| G0
G0-->G["Default Page"]

Nginx manages traffic entry points at the application layer. The entire host only needs to have ports 443 and SSH open. Additionally, various modules have been configured for obfuscation. For “non-standard requests,” everything appears as normal pages, and CDN for WS protocol works perfectly.

Thanks to Nginx’s excellent performance and support for the HTTP protocol, all requests can utilize HTTP2.

ALL HTTP2

YouTube’s speed compared to direct Trojan connection doesn’t show any significant difference.

Youtube Speed

Let the professionals handle professional matters 😎.

Lastly, I’d like to provide a solution to the issue of “getting the real client IP as 127.0.0.1.” In fact, this issue could be the subject of a standalone article. For the sake of readability, I’m including it here.

This was an oversight on my part. Since my web service is my own blog, it is initially behind a CDN, which already includes the real client IP in the header. I adapted the setup to account for this scenario, so the issue didn’t surface.

However, when I removed the CDN, I found that I was indeed getting the real client IP as 127.0.0.1. Why does this happen?

Firstly, let me explain two concepts:

First, real client IP is usually obtained from the remote_addr or http_x_forwarded_for headers in HTTP requests.

Regarding remote_addr, it’s not specific to HTTP. It’s determined when the TCP connection is established. Therefore, it’s not easily spoofed.

graph LR
A["Client"] -->|TCP: remote_addr| B("TCP")
B --> C["HTTP"]

On the other hand, http_x_forwarded_for is specific to HTTP. Every time a request passes through a layer-7 proxy, the proxy server appends its IP to the header. So, it typically looks like this: the real user IP (remote_addr), followed by the IP of proxy server 1, the IP of proxy server 2, and so on.

graph LR
A["Client"] -->|TCP: remote_addr| B("TCP")
B --> |HTTP1: http_x_forwarded_for| C["HTTP Proxy 1"]
C --> |HTTP2: http_x_forwarded_for| D["HTTP Proxy 2"]
D --> |HTTP3: http_x_forwarded_for| E["HTTP Proxy n"]

Second, there’s the matter of 4th-layer TCP load balancing/proxy mechanisms.

When Nginx forwards TCP requests directly, it essentially establishes a new TCP connection for each backend service. This direct forwarding at the TCP layer ensures data integrity and higher performance. However, since remote_addr is determined at the time of TCP connection establishment, each backend service ends up seeing Nginx’s IP.

graph LR
A["Client"] -->|TCP1: remote_addr| B("Nginx")
B -->C{"dispatch"}
C -->|TCP2: remote_addr| D["Trojan"]
C -->|TCP3: remote_addr| E["V2Ray"]
C -->|TCP4: remote_addr| F["Web"]

This is an inherent issue when dealing with 4th-layer proxy software.

I checked the official Nginx documentation and found that Nginx Plus offers solutions for this. However, these solutions are available in the free version as well. There are two options:

Option 1: Communicate using the Proxy Protocol.

I mentioned earlier that this real client IP issue exists with various 4th-layer proxy software. So, a special communication protocol called the Proxy Protocol was designed to pass the real client IP.

The Proxy Protocol, developed by Willy Tarreau, the author of HAProxy, in 2010, is an Internet protocol that adds a small header to TCP to

conveniently pass client information (protocol stack, source IP, destination IP, source port, destination port, etc.) when network conditions are complex and it’s necessary to obtain the real client IP. Essentially, it inserts a packet carrying the original connection’s quadruple information into the connection after the three-way handshake.

Nginx also supports the Proxy Protocol, so you just need to configure Proxy Protocol support on both the forwarding side and the receiving side to use it.

Introducing the proxy_protocol configuration at the forwarding layer.

stream {
    # If you wish to log proxy protocol information, configure the necessary variables as needed. $proxy_protocol_addr represents the real client IP.
    # More details at: https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol
    # log_format basic '$proxy_protocol_addr - $remote_user [$time_local] '
    #                  '$protocol $status $bytes_sent $bytes_received '
    #                  '$session_time';
    server {
        listen 443 reuseport;
        listen [::]:443 reuseport;
        proxy_pass  $backend_name;
        proxy_protocol on;
        ssl_preread on;
    }
}

Simultaneously, the backend service also needs to be configured to accept proxy_protocol within the listen rule.

http {
    # If you wish to log proxy protocol information, configure the necessary variables as needed. $proxy_protocol_addr represents the real client IP.
    # More details at: https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol
    # log_format combined '$proxy_protocol_addr - $remote_user [$time_local] '
    #                     '"$request" $status $body_bytes_sent '
    #                     '"$http_referer" "$http_user_agent"';
    server {
        listen 10242 proxy_protocol; # Note the addition of the protocol type here
        server_name  vmess.com.chengxiaobai;
        # ...  
        # Omitting some configuration examples
        # ...
    }
}

However, this approach has a drawback: once the receiving end is configured for the proxy protocol, it no longer supports non-proxy protocol requests.

The receiving end of the Proxy Protocol must begin processing connection data only after receiving a complete and valid Proxy Protocol header. Therefore, for the same listening port on the server, connections that are not compatible with the Proxy Protocol are not supported. If the server receives a data packet that does not conform to the Proxy Protocol format as the first packet, the server will terminate the connection directly.

It’s important to note that Trojan does not support the Proxy Protocol. So, once Nginx uses the Proxy Protocol for forwarding, an intermediate layer is needed to help Trojan remove the protocol. This ensures that other services can obtain the real client IP through the Proxy Protocol and that the traffic forwarded to Trojan is handled correctly.

stream {
    map $ssl_preread_server_name $backend_name {
      # Direct Trojan traffic to the intermediate layer: proxy_trojan
      trojan.com.chengxiaobai proxy_trojan;
      # ...  
      # Omitting some configuration examples
      # ...
    }
    upstream trojan {
        server 127.0.0.1:10241;
    }
    upstream proxy_trojan {
        server 127.0.0.1:10249;
    }
    server {
        listen 443 reuseport;
        listen [::]:443 reuseport;
        proxy_pass  $backend_name;
        proxy_protocol on;
        ssl_preread on;
    }
    # This server is the intermediate layer responsible for helping Trojan remove the proxy protocol
    # The original upstream trojan configuration remains unchanged
    server {
        listen 10249 proxy_protocol;
        proxy_pass  trojan;
    }
}

Option 2: Use the proxy_bind syntax.

user root; # Note the permission requirement here

# Requires additional OS-level configuration!

# https://WWW.kernel.org/doc/Documentation/networking/tproxy.txt

stream {
    server {
        listen 443 reuseport;
        listen [::]:443 reuseport;
        proxy_bind $remote_addr transparent; # This is the new addition
        proxy_pass  $backend_name;
        ssl_preread on;
    }
}

This approach doesn’t modify the communication protocol but requires Nginx itself to run with root user privileges. It also necessitates configuring the local routing table on the backend service machine to perform IP redirection.

The actual operation works as follows: this configuration changes the source address of TCP connections to the real client IP rather than Nginx’s IP. However, when the backend service returns data, there can be issues because the real client hasn’t established a communication link with the backend service. Therefore, routing must be configured on the backend service to forward request-response traffic to the Nginx machine. Additionally, Nginx machine needs to have local IP interception rules to redirect this traffic to the Nginx service, completing a form of “traffic hijacking.”

To understand this better, think of it as a Man-in-the-Middle (MITM) attack, but at a lower level than the 7-layer HTTP protocol, as it deals with the 4-layer TCP protocol.

In summary, I don’t recommend using option 2 due to its security and operational challenges. V2Ray and Trojan currently don’t rely on the real client IP, so if you strongly depend on the real client IP (more likely in web services?), I suggest using option 1.

References: