LinkerD retry failing

I am new to LinkerD and I can’t get retries to work. I have configured a ServiceProfile:

apiVersion: linkerd.io/v1alpha2
kind: ServiceProfile
metadata:
  creationTimestamp: null
  name: report.test-linkerd.svc.cluster.local
  namespace: test-linkerd
spec:
  routes:
    - name: All GET Requests
      condition:
        method: GET
        pathRegex: ".*"
      isRetryable: true
    - name: All POST Requests
      condition:
        method: POST
        pathRegex: ".*"
      isRetryable: true
  retryBudget:
    retryRatio: 0.2
    minRetriesPerSecond: 10
    ttl: 10s

I have a service profile that has two routes, one for GET and one for POST. I have a retry budget set for the service profile.
When I try to make a condition for a retry (killing a downstream pod), I see that the retries are not happening.

I started to investigate,

➜  ~ linkerd viz routes --to deploy/report -n test-linkerd -o wide deploy/api-gateway
ROUTE                        SERVICE   EFFECTIVE_SUCCESS   EFFECTIVE_RPS   ACTUAL_SUCCESS   ACTUAL_RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
All GET Requests    portfolio-report                   -               -                -            -             -             -             -
All POST Requests   portfolio-report                   -               -                -            -             -             -             -
[DEFAULT]           portfolio-report                   -               -                -            -             -             -             -

And I see that the routes are not being matched. I have checked the logs of the api-gateway and I see that the requests are being made to the report service.
Then I ran the following command to check the metrics of the report service:

➜ ~ linkerd diagnostics proxy-metrics -n test-linkerd deploy/report | grep route_response_total # HELP route_response_total Total count of HTTP responses. # TYPE route_response_total counter route_response_total{direction="inbound",dst="report.test-linkerd.svc.cluster.local:80",rt_route="All POST Requests",status_code="200",classification="success",grpc_status="",error=""} 5 route_response_total{direction="inbound",dst="report.test-linkerd.svc.cluster.local:8090",rt_route="All GET Requests",status_code="200",classification="success",grpc_status="",error=""} 9 route_response_total{direction="inbound",dst="report.test-linkerd.svc.cluster.local:8090",rt_route="All GET Requests",status_code="304",classification="success",grpc_status="",error=""} 44

And I see that the report service is receiving requests with the routes All GET Requests and All POST Requests.
Additionally, in the dashboard, I do not see any requests from the api-gateway to the report service. They do show as “meshed”, but there is no green bar or metrics.
The only other clue I have found is the logs of the linkerd proxy container:

{"timestamp":"[  1355.773887s]","level":"INFO","fields":{"message":"Connection closed","error":"connection closed before message completed","client.addr":"10.62.71.212:35084","server.addr":"10.62.68.10:8080"},"target":"linkerd_app_core::serve","spans":[{"name":"inbound"}],"threadId":"ThreadId(1)"}

Am I missing something in the configuration of the service profile?
Am I misunderstanding how retries work in LinkerD?
Any help would be appreciated. Thanks!
Additional Info:

Client version: stable-2.14.10
Server version: stable-2.14.10

linkerd check and linkerd viz check come back all good.

Your ServiceProfile works but I don’t think it can pass your test scenario because the retry budget is depleted by the time the pod has restarted.

To verify that it works in general, I have taken a copy of your “All GET Requests” route and created a full example.

kind: Namespace
apiVersion: v1
metadata:
  name: retries-test
  annotations:
    linkerd.io/inject: enabled
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: random-failures-api
  namespace: retries-test
  labels:
    app: random-failures-api
spec:
  selector:
    matchLabels:
      app: random-failures-api
  template:
    metadata:
      labels:
        app: random-failures-api
    spec:
      containers:
        - name: random-failures-api
          image: "ghcr.io/stzov/random-failing-api:rel-v0.1.6"
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 5678
              protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: random-failures-api
  namespace: retries-test
  labels:
    app: random-failures-api    
spec:
  selector:
      app: random-failures-api
  ports:
  - protocol: TCP
    port: 5678
    targetPort: 5678
    appProtocol: http
---
apiVersion: linkerd.io/v1alpha2
kind: ServiceProfile
metadata:
  name: random-failures-api.retries-test.svc.cluster.local
  namespace: retries-test
spec:
  routes:
  - name: All GET Requests
    condition:
      method: GET
      pathRegex: ".*"
    isRetryable: true
    responseClasses:
    - condition:
        status:
          min: 400
          max: 599
      isFailure: true
    timeout: 60s
  retryBudget:
    retryRatio: 0.2
    minRetriesPerSecond: 10
    ttl: 10s

The random-failing-api image I used randomly generates occasional 4xx and 5xx errors when deployed in a pod.

Once you deploy the above you’ll need a curl enabled pod and try running:

while true; do curl -v random-failures-api.retries-test.svc.cluster.local:5678/get ; sleep 1 ; done

You will notice that you only get HTTP/1.1 200 OK responses, even though the relevant container returns random errors.

If you remove the responseClasses configuration I added to your route then by default linkerd will consider only 5xx as an error so if you try to curl again you will occasionally get a 4xx response.

So this shows that linkerd is doing what it is supposed to do but it’s failing to cover the scenario you described. To be honest it would be nice if it was able to somehow queue the requests until the pod is up but I don’t think this is in scope.

An easy solution for your scenario would be to have a multiple replicas for your pod. In that case, when at least one pod is alive you will not get an error.

The source code for the image I used can be found here

Filed under: Kiến thức lập trình - @ 20:18

Thẻ: linkerd

Thiết kế website giá rẻ

Danh mục

LinkerD retry failing