Proposal - improve logging of calls sending data and a user interface for reading the log

[jaapjansma]

Problem and motivation behind this proposal

At one of my clients they have observed that at certain times of the day CiviCRM is slow and unresponsive. If at that time someone signs up for the newsletter on the website. The data does not always ends up in CiviCRM. What we want is a user interface in CiviCRM which reads the CiviProxy log and which gives the ability to repost a certain request.

When donations and newsletter signups are happening on the website they can also see and check the CiviCRM call log in Drupal. However this client also has various agencies calling civicrm api's and sending in data.

Considerations to keep in mind

The following requirements we have for looking for a solution

• Lightweight CiviProxy is a lightweight solution (it has no database for example). We still want to keep this.
• Performance when the trafic is high right now CiviProxy is also performing very good (it is affected by the performance of CiviCRM though)
• Data retention and security we do not want that data stays forever on the CiviProxy. This expose a little security risk

Possible solution

1. Every call which sends data to CiviCRM should be logged. The data logged is in the incoming data as well as the response. And some meta data such as date and time of the call.
2. CiviCRM pulls this log (daily) and stores it in the CiviCRM database. When the log is pulled the log on the CiviProxy is cleared
3. CiviCRM has a user interface listing all the log entries and to replay and alter the parameters of a failed call

Challenge of reading and writing to a log file

The challenge is to have a solution which writes very fast to a log file. But also when CiviCRM pulls the data the data first written should read first (FIFO) but also cleared. Writing to disk is usually very slow. An alternative is to store the data in separate queue software such as Redis however that means the Proxy server needs another software installed (and maintained by the server administrator)

Implementation with composer

Maybe it is good to implement this as a plugin to CiviProxy. And have CiviProxy to have composer.json so that one can install CiviProxy with composer and install the additional plugins as well with composer.

[jensschuppe]

First thoughts:

• FIFO logging could be done by a tight rotation schedule, i.e. CiviCRM always pulls the latest log file entirely so that deletion only has to be done on a per-file basis instead of deleting contents from files
• As for the CiviCRM side, this could be an extension that can pull API requests from configurable sources and formats (not only CiviProxy log files) and thus wouldn't have to be a dependency or depend on CiviProxy at all
• Logging on the proxy could be made pluggable (so that external software can be used)
• Composer 👍!

[jaapjansma]

FIFO logging could be done by a tight rotation schedule, i.e. CiviCRM always pulls the latest log file entirely so that deletion only has to be done on a per-file basis instead of deleting contents from files

Just discussed this during a call. We need a setting of how many calls per log file and the time frame per log file. For example. max 100 calls in a log file and the time frame is 3 hours. This means that CiviCRM can only read the log file after the 3 hours or after 100 recorded calls.

As for the CiviCRM side, this could be an extension that can pull API requests from configurable sources and formats (not only CiviProxy log files) and thus wouldn't have to be a dependency or depend on CiviProxy at all

Very good idea!

Logging on the proxy could be made pluggable (so that external software can be used)

Yes, indeed and that is the plan and composer helps for that.