How do I prevent large InnoDB table writes from hanging the database?

  Kiến thức lập trình

I have a Percona 8 MySQL Server, running via Docker, acting as the backing data store for a heavily used service.
On the hour, a script will run that reads a value from a virtual column of approximately 1.7 million rows (Table size estimate of 2.3 GiB) and inserts this particular value and the associated key data into another table that the system otherwise only reads from. The virtual column is a JSON lookup json_extract(jsonData, '$.root.interestingValue') and has the GENERATED flag.
The idea behind this is to put less pressure on the table that is considered live and updated regularly, when users are only interested in a few specific values and retrieving the absolute latest value is not required.

The query is as follows (with tables/columns renamed)

CREATE TEMPORARY TABLE t1_cache_temp
SELECT
  t2.id as uid,
  t3.displayText as dt,
  t2.virtualColumn as interestingValue
FROM liveTable t2
JOIN otherLiveTable t3 on t2.id = t3.id;

TRUNCATE TABLE t1_cache;

INSERT INTO t1_cache
SELECT uid, dt, interestingValue FROM t1_cache_temp;

DROP TEMPORARY TABLE t1_cache_temp;

The total time taken by this script is 47 seconds.

Whilst reading from the InnoDB table and writing to the temporary table is fine, writing it to the other InnoDB table causes all other operations on the database to hang after the first few seconds.
I narrowed it down to the INSERT INTO t1_cache... statement by executing each statement individually.

I have also tried writing to new and completely unused tables (also using InnoDB), which gives the same result. At the time of performing the last test, only 2% of the permitted Max Connections are in use. Also, only 48% of the InnoDB Buffer Pool was in use.

If I switch the table to MyISAM, everything works normally, with no interruption or hanging. Additionally, if I remove the temporary table, it causes the same issue regardless of InnoDB or MyISAM engine.

Whilst using MyISAM is probably the better option anyway, what could be causing this? Is there anything I can do about it at a configuration level?

LEAVE A COMMENT