Ticket comments are incomplete and inaccurate

Hey so, I've been using the Zendesk REST API for email channels to retrieve ticket comments and it's been fairly straightforward so far; however, now I'm working on other channels like mobile_sdk, chat and instagram_dm and I'm running into a whole host off issues accessing the comments.

I'm facing 2 main issues

1. There is a major delay from when the ticket has new comments and when it's reflected via the API endpoints /tickets/[ticketID]/comments
2. The format for channels like instagram_dm is completely broken

For example, I've been using ZAF as well and I pass the ticket via ZAF to my backend and it has a conversations attribute which is very neatly formatted. 

Where as in the REST API for comments, the author_id is set to -1 and the data in the plain_body, body etc is a jumble of messages from both the customer and agent.

Here's a sample from the ZAF client

[
  {
    "timestamp": "2025-01-08T10:07:39Z",
    "author": {
      "id": "24248831486621",
      "avatar": "https://assets.zendesk.com/images/2016/default-avatar-80.png",
      "name": "AG",
      "role": "end-user"
    },
    "channel": { "name": "instagram_dm" },
    "message": {
      "content": "When is your next studio course",
      "contentType": "text/plain"
    }
  },
  {
    "timestamp": "2025-01-09T06:35:48Z",
    "author": {
      "id": "22986182800797",
      "avatar": "https://redact/system/photos/22993189114653/profile.jpg",
      "name": "VR",
      "role": "admin"
    },
    "channel": { "name": "instagram_dm" },
    "message": {
      "content": "Hello AG 😊 Our next studio course is the Lord Balaji - 2.5 Months Course starting on January 13th, 2025. You can find more details here (https://www.redact/products/lord-balaji-2-5-months-course).",
      "contentType": "text/plain"
    }
  },
  {
    "timestamp": "2025-01-09T06:36:47Z",
    "author": {
      "id": "22986182800797",
      "avatar": "https://redact/system/photos/22993189114653/profile.jpg",
      "name": "VR",
      "role": "admin"
    },
    "channel": { "name": "instagram_dm" },
    "message": {
      "content": "How much does it cost?",
      "contentType": "text/plain"
    }
  }
]

Here's the same sample using the REST endpoint ticket/comments

{
    "id": 24274457817117,
    "type": "Comment",
    "author_id": 24248831486621,
    "body": "Conversation with AG\n\nURL: None",
    "html_body": "<div class=\"zd-comment\" dir=\"auto\"><p dir=\"auto\">Conversation with AG</p>\n\n<p dir=\"auto\">URL: None</p></div>",
    "plain_body": "Conversation with AG \n\n URL: None",
    "public": false,
    "attachments": [],
    "audit_id": 24274457816861,
    "via": {
        "channel": "instagram_dm",
        "source": {
            "from": {},
            "to": {},
            "rel": null
        }
    },
    "created_at": "2025-01-08T10:07:41Z",
    "metadata": {
        "system": {},
        "custom": {}
    }
},
{
    "id": 24274823185565,
    "type": "Comment",
    "author_id": -1,
    "body": "(15:37:39) AG: When is your next studio course",
    "html_body": "<div class=\"zd-comment\" dir=\"auto\"><p dir=\"auto\">(15:37:39) AG: When is your next studio course</p></div>",
    "plain_body": "(15:37:39) AG: When is your next studio course",
    "public": true,
    "attachments": [],
    "audit_id": 24274823185309,
    "via": {
        "channel": "chat_transcript",
        "source": {
            "from": {},
            "to": {},
            "rel": null
        }
    },
    "created_at": "2025-01-08T10:17:41Z",
    "metadata": {
        "system": {},
        "custom": {}
    }
},
{
    "id": 24297177864733,
    "type": "Comment",
    "author_id": -1,
    "body": "(12:05:48) VR: Hello AG 😊 Our next studio course is the Lord Balaji - 2.5 Months Course starting on January 13th, 2025. You can find more details here (https://redact.com/products/lord-balaji-2-5-months-course).\n(12:06:47) VR: How much does it cost?",
    "html_body": "<div class=\"zd-comment\" dir=\"auto\"><p dir=\"auto\">(12:05:48) VR: Hello AG 😊 Our next studio course is the Lord Balaji - 2.5 Months Course starting on January 13th, 2025. You can find more details here (<a href=\"https://redact.com/products/lord-balaji-2-5-months-course\" target=\"_blank\" rel=\"nofollow noreferrer\">https://redact.com/products/lord-balaji-2-5-months-course</a>).\n<br>(12:06:47) VR: How much does it cost?</p></div>",
    "plain_body": "(12:05:48) VR: Hello AG 😊 Our next studio course is the Lord Balaji - 2.5 Months Course starting on January 13th, 2025. You can find more details here (https://redact.com/products/lord-balaji-2-5-months-course).\n\n(12:06:47) VR: How much does it cost?",
    "public": true,
    "attachments": [],
    "audit_id": 24297209391005,
    "via": {
        "channel": "chat_transcript",
        "source": {
            "from": {},
            "to": {},
            "rel": null
        }
    },
    "created_at": "2025-01-09T06:37:09Z",
    "metadata": {
        "system": {},
        "custom": {}
    }
}

As you can see in the REST API, since the author_id is -1 I have no way of identifying who the author is. Moreover, the messages from both the customer and agent are clubbed into the same comment.

Does anyone know if there's a way to get the format that the ZAF client provides using the REST API? Since this is proving to be quite a challenge to get an accurate format for a ticket conversation.