health check: malformed request [http vs https]

[schepens83]

Hi all

I'm trying to deploy a RoR app to a digitalocean droplet. The healthcheck seems to fail because it is doing so with an https connection while it expects http. The rails app is quite new with little config. Deploying to server was my first step after creating a quick landingpage. Hopefully someone can point me in the right direction.

Addition to production.rb on recommendation of claude. (although kamal 2.0 is too new for claude to be much help)

...
config.hosts.clear
config.force_ssl = true
config.ssl_options = { redirect: { exclude: ->(request) { request.headers["X-Forwarded-Proto"] == "https" } } }

deploy.yml

# Name of your application. Used to uniquely configure containers.
service: detuintovenaar

# Name of the container image.
image: schepenss/detuintovenaar

# Deploy to these servers.
servers:
  web:
    - 188.166.5.214
  # job:
  #   hosts:
  #     - 192.168.0.1
  #   cmd: bin/jobs

# Enable SSL auto certification via Let's Encrypt and allow for multiple apps on a single web server.
# Remove this section when using multiple web servers and ensure you terminate SSL at your load balancer.
#
# Note: If using Cloudflare, set encryption mode in SSL/TLS setting to "Full" to enable CF-to-app encryption. 
proxy: 
  ssl: true
  host: www.detuintovenaar.com
  # Proxy connects to your container on port 80 by default.
  app_port: 3000
  healthcheck:
    path: /up
    interval: 2
    timeout: 2

# Credentials for your image host.
registry:
  # Specify the registry server, if you're not using Docker Hub
  # server: registry.digitalocean.com / ghcr.io / ...
  username: schepenss

  # Always use an access token rather than real password (pulled from .kamal/secrets).
  password:
    - KAMAL_REGISTRY_PASSWORD

# Configure builder setup.
builder:
  arch: amd64

# Inject ENV variables into containers (secrets come from .kamal/secrets).
#
env:
  secret:
    - RAILS_MASTER_KEY

error message from kamal:

...
INFO [d7ec196e] Running docker container ls --all --filter name=^detuintovenaar-web-826f0cd20e46c7a950cfb09dc824bffaa36fb1a6$ --quiet on 188.166.5.214
  INFO [d7ec196e] Finished in 0.104 seconds with exit status 0 (successful).
  INFO [ca39320f] Running docker exec kamal-proxy kamal-proxy deploy detuintovenaar-web --target="5c71ded028da:3000" --host="www.detuintovenaar.com" --tls --deploy-timeout="30s" --drain-timeout="30s" --health-check-interval="2s" --health-check-timeout="2s" --health-check-path="/up" --buffer-requests --buffer-responses --log-request-header="Cache-Control" --log-request-header="Last-Modified" --log-request-header="User-Agent" on 188.166.5.214
 ERROR Failed to boot web on 188.166.5.214
  INFO First web container is unhealthy on 188.166.5.214, not booting any other roles
  INFO [40829406] Running docker container ls --all --filter name=^detuintovenaar-web-826f0cd20e46c7a950cfb09dc824bffaa36fb1a6$ --quiet | xargs docker logs --timestamps 2>&1 on 188.166.5.214
  INFO [40829406] Finished in 0.109 seconds with exit status 0 (successful).
 ERROR 2024-11-07T22:07:58.238924651Z => Booting Puma
2024-11-07T22:07:58.238981687Z => Rails 7.2.2 application starting in production
2024-11-07T22:07:58.238985696Z => Run `bin/rails server --help` for more startup options
2024-11-07T22:08:00.077998601Z Puma starting in single mode...
2024-11-07T22:08:00.078235488Z * Puma version: 6.4.3 (ruby 3.3.0-p0) ("The Eagle of Durango")
2024-11-07T22:08:00.078350738Z *  Min threads: 3
2024-11-07T22:08:00.078421433Z *  Max threads: 3
2024-11-07T22:08:00.078495053Z *  Environment: production
2024-11-07T22:08:00.078574831Z *          PID: 1
2024-11-07T22:08:00.079517278Z * Listening on http://0.0.0.0:3000
2024-11-07T22:08:00.083838644Z Use Ctrl-C to stop
2024-11-07T22:08:01.351095457Z 2024-11-07 22:08:01 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
2024-11-07T22:08:03.347295619Z 2024-11-07 22:08:03 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
2024-11-07T22:08:05.345906593Z 2024-11-07 22:08:05 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
2024-11-07T22:08:07.347635583Z 2024-11-07 22:08:07 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
2024-11-07T22:08:09.348144785Z 2024-11-07 22:08:09 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
2024-11-07T22:08:11.347352105Z 2024-11-07 22:08:11 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
2024-11-07T22:08:13.346503061Z 2024-11-07 22:08:13 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
2024-11-07T22:08:15.349473389Z 2024-11-07 22:08:15 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
2024-11-07T22:08:17.345912512Z 2024-11-07 22:08:17 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
2024-11-07T22:08:19.352781915Z 2024-11-07 22:08:19 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
2024-11-07T22:08:21.355998119Z 2024-11-07 22:08:21 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
2024-11-07T22:08:23.345571219Z 2024-11-07 22:08:23 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
2024-11-07T22:08:25.348756797Z 2024-11-07 22:08:25 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
  INFO [9ccf2d83] Running docker container ls --all --filter name=^detuintovenaar-web-826f0cd20e46c7a950cfb09dc824bffaa36fb1a6$ --quiet | xargs docker inspect --format '{{json .State.Health}}' on 188.166.5.214
  INFO [9ccf2d83] Finished in 0.109 seconds with exit status 0 (successful).
 ERROR null
  INFO [19a5b471] Running docker container ls --all --filter name=^detuintovenaar-web-826f0cd20e46c7a950cfb09dc824bffaa36fb1a6$ --quiet | xargs docker stop on 188.166.5.214
  INFO [19a5b471] Finished in 0.309 seconds with exit status 0 (successful).
  Finished all in 65.7 seconds
Releasing the deploy lock...
  Finished all in 67.8 seconds
  ERROR (SSHKit::Command::Failed): Exception while executing on host 188.166.5.214: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy

[tuladhar]

My hunch is the issue with following SSL config you have in production.rb, try removing and see if it helps.

config.force_ssl = true
config.ssl_options = { redirect: { exclude: ->(request) { request.headers["X-Forwarded-Proto"] == "https" } } }

The healthcheck is at /up endpoint, and it looks like there's redirect to HTTPS going on.

[theaboutbox]

I had this issue and I struggled for hours to get past it. Here are all of the things that I needed to do, I don't know if they are all necessary but I wound up getting things to work:

Upgrade to Rails 8

I updated the gem in my Gemfile, and I ran rails app:update to update the rest of the application. Whenever prompted, I replaced files with the Rails 8 version. The relevant section of my production.rb looks like this, per the default:

  # Assume all access to the app is happening through a SSL-terminating reverse proxy.
  config.assume_ssl = true

  # Force all access to the app over SSL, use Strict-Transport-Security, and use secure cookies.
  config.force_ssl = true

  # Skip http-to-https redirect for the default health check endpoint.
  # config.ssl_options = { redirect: { exclude: ->(request) { request.path == "/up" } } }

Install Thruster and update the Dockerfile

bundle add thruster to install Thruster. Then I updated the end of the Dockerfile thusly:

EXPOSE 80
CMD ["./bin/thrust", "./bin/rails", "server"]

Go back to defaults in deploy.yml

proxy: 
  ssl: true
  host: [YOUR DNS NAME]
  # Proxy connects to your container on port 80 by default.
  # app_port: 3000

Commit and push changes

At this point, I needed to commit and push changes so that Kamal would update the Dockerfile.

Finish setup process

Finally, I was able to run kamal setup and it completed properly.

Configure DNS correctly

I am using Cloudflare as my DNS provider. It has a proxy option for the DNS record but I had problems with infinite redirects, so I updated the configuration to look like this:

Conclusion

I don't know if all of these steps are necessary, but I had to do all of these things in order to get the deploy to work. I don't understand yet why I needed Thruster + Rails 8 to get rid of the malformed request errors but I had to do both of those things.

[schepens83]

I ended up with this config in my production.rb

config.hosts.clear

  config.hosts << "[didacted]"
  config.hosts << "[didacted]"
  config.hosts << /\h+/  # For docker container IDs

  # Force SSL but exclude internal requests
  config.force_ssl = true
  config.ssl_options = {
    redirect: {
      exclude: ->(request) {
        request.path == "/up" ||
        request.headers["X-Forwarded-Proto"] == "https"
      }
    }
  }

What i also did was remove all images and containers from the server before deploying again. After that it worked for me. Don't know what did the trick ¯_(ツ)_/¯

[shiny]

The issue I encountered is that I am deploying an accessory with a third-party image, so I cannot modify the source code. Since user authentication is enabled, the HTTP status returned is not 200, which causes the health check to fail.
It seems kamal is not able to set the expected status code.

[andres57706]

Hey guys I was struggling with this matter yesterday with an "old" rails 7 app all you guys shared help me a lot, I will give my solution in order to help others in future

So I look this comment: #1204 (comment) (same problem i had)

Then this one: #1204 (comment) (the possible solution)

For those with rails apps prior to 8 version, its not necessary to upgrade (if you upgrade if you are not really sure if you already have a funcional app or platform making money or serving users, so structure your upgrade plan)

I look into these files on rails repo: https://github.com/rails/rails/blob/main/railties/lib/rails/generators/rails/app/templates/config/environments/production.rb.tt

Also and very important read your proxy logs with kamal proxy logs and kamal app logs in order to figure it out, maybe its something different.

If this is your case as the initial comment then use the production.rb.tt form rails 8 generator and exclude only the '/up' route:

• https://github.com/rails/rails/blob/604e8e057020d3ae522f674e3fdcaf4dea160d31/railties/lib/rails/generators/rails/app/templates/config/environments/production.rb.tt#L38

Check your config/deploy.yml in case you have another rails server PORT, or set the env. Also check your Dockerfile at the very end rails 7 app EXPOSE 3000 port so keep this in mind on your kamal's proxy setup app_port: 3000.

Thats every thing you need, you don need to upgrade (at least now) and you don't have to install thruster unless you really needed.

I hope it helps for those with apps prior to rails 8 🚀