Partitioning with external script for mysql/mariadb database

Preparation

If your database stored in one file (example /var/lib/mysql/ibdata), you need activate innodb_file_per_table option in mysql config.

Algorithm:

Stop services, that uses database; Backup database; Delete database; Stop mysql server; In file /etc/mysql/my.cnf delete option innodb_data_file_path and add innodb_file_per_table; Delete all files like /var/lib/mysql/ibdata, /var/lib/mysql/ib_logfile0, /var/lib/mysql/ib_logfile1; Start mysql server; Create database; Restore data from backup; Now we must see files with extension .ibd for every table Start services.

We need to split these tables in zabbix database: history,history_uint,history_str,history_text,history_log,trends,trends_uint

Disable housekeeping in web for history and trends Administration -> General -> Housekeeping.

If we want disable housekeeping for all tables, change HousekeepingFrequency to 0 in /etc/zabbix/zabbix_server.conf.

SHOW ENGINES; | INSTALL SONAME 'ha_blackhole'; or INSTALL PLUGIN ALTER TABLE housekeeper ENGINE = BLACKHOLE; SHOW GLOBAL VARIABLES LIKE 'event_scheduler'; SET GLOBAL event_scheduler = ON; You should also put a line in the 'my.cnf' file like "event_scheduler=ON" in case of reboot.

Now it's time to start partition for each table. As partitioning is usually performed for a database with existing historical data - for every table you must specify partitions starting from a minimum value of the clock field and up to the current moment (day, month) of tables to be partitioned. The minimum value of the clock in a table can be found out by a query like this:

SELECT FROM_UNIXTIME(MIN(clock)) FROM history_uint;

Example of commands for manual init/add/delete partition, just in case:

ALTER TABLE `history` PARTITION BY RANGE (clock) (PARTITION p2011_11_24 VALUES LESS THAN (UNIX_TIMESTAMP("2011-11-25 00:00:00")));
ALTER TABLE `history` ADD PARTITION p2011_10_23 VALUES LESS THAN (UNIX_TIMESTAMP("2011-10-24 00:00:00")) ENGINE = InnoDB;
ALTER TABLE `history` DROP PARTITION p2011_06;
full sql
        
ALTER TABLE `trends` PARTITION BY RANGE( clock ) (
PARTITION p2019_08 VALUES LESS THAN (UNIX_TIMESTAMP("2019-09-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_09 VALUES LESS THAN (UNIX_TIMESTAMP("2019-10-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_10 VALUES LESS THAN (UNIX_TIMESTAMP("2019-11-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_11 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12 VALUES LESS THAN (UNIX_TIMESTAMP("2020-01-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2020_01 VALUES LESS THAN (UNIX_TIMESTAMP("2020-02-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2020_02 VALUES LESS THAN (UNIX_TIMESTAMP("2020-03-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2020_03 VALUES LESS THAN (UNIX_TIMESTAMP("2020-04-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2020_04 VALUES LESS THAN (UNIX_TIMESTAMP("2020-05-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2020_05 VALUES LESS THAN (UNIX_TIMESTAMP("2020-06-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2020_06 VALUES LESS THAN (UNIX_TIMESTAMP("2020-07-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2020_07 VALUES LESS THAN (UNIX_TIMESTAMP("2020-08-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2020_08 VALUES LESS THAN (UNIX_TIMESTAMP("2020-09-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2020_09 VALUES LESS THAN (UNIX_TIMESTAMP("2020-10-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2020_10 VALUES LESS THAN (UNIX_TIMESTAMP("2020-11-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2020_19 VALUES LESS THAN (UNIX_TIMESTAMP("2020-12-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2020_12 VALUES LESS THAN (UNIX_TIMESTAMP("2021-01-01 00:00:00")) ENGINE = InnoDB);

ALTER TABLE `trends_uint` PARTITION BY RANGE( clock ) (
-//-

ALTER TABLE `history_uint` PARTITION BY RANGE( clock ) (
PARTITION p2019_10_19 VALUES LESS THAN (UNIX_TIMESTAMP("2019-11-20 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_11_20 VALUES LESS THAN (UNIX_TIMESTAMP("2019-11-21 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_11_21 VALUES LESS THAN (UNIX_TIMESTAMP("2019-11-22 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_11_22 VALUES LESS THAN (UNIX_TIMESTAMP("2019-11-23 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_11_23 VALUES LESS THAN (UNIX_TIMESTAMP("2019-11-24 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_11_24 VALUES LESS THAN (UNIX_TIMESTAMP("2019-11-25 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_11_25 VALUES LESS THAN (UNIX_TIMESTAMP("2019-11-26 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_11_26 VALUES LESS THAN (UNIX_TIMESTAMP("2019-11-27 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_11_27 VALUES LESS THAN (UNIX_TIMESTAMP("2019-11-28 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_11_28 VALUES LESS THAN (UNIX_TIMESTAMP("2019-11-29 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_11_29 VALUES LESS THAN (UNIX_TIMESTAMP("2019-11-30 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_11_30 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-01 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_01 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-02 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_02 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-03 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_03 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-04 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_04 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-05 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_05 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-06 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_06 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-07 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_07 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-08 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_08 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-09 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_09 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-10 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_10 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-11 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_11 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-12 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_12 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-13 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_13 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-14 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_14 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-15 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_15 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-16 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_16 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-17 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_17 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-18 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_18 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-19 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_19 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-20 00:00:00")) ENGINE = InnoDB,
PARTITION p2019_12_20 VALUES LESS THAN (UNIX_TIMESTAMP("2019-12-21 00:00:00")) ENGINE = InnoDB);

ALTER TABLE `history_log` PARTITION BY RANGE( clock ) (
-//-

ALTER TABLE `history_text` PARTITION BY RANGE( clock ) (
-//-

ALTER TABLE `history` PARTITION BY RANGE( clock ) (
-//-

ALTER TABLE `history_str` PARTITION BY RANGE( clock ) (
-//-
        
    

How to use script

You can use another script.

yum install -y perl-Sys-Syslog
yum install -y perl-DateTime-TimeZone
chmod +x /etc/cron.daily/zabbix-partition
/etc/cron.daily/zabbix-partition

In code u must choose period of history and version of mysql.

Expand script code
        
#!/usr/bin/perl

use strict;
use Data::Dumper;
use DBI;
use Sys::Syslog qw(:standard :macros);
use DateTime;
use POSIX qw(strftime);

openlog("mysql_zbx_part", "ndelay,pid", LOG_LOCAL0);

my $db_schema = 'zabbixdb';
my $dsn = 'DBI:mysql:'.$db_schema.':mysql_socket=/var/lib/mysql/mysql.sock';
my $db_user_name = 'root';
my $db_password = 'Nxtt2019';
my $tables = {  'history' => { 'period' => 'day', 'keep_history' => '30'},
        'history_log' => { 'period' => 'day', 'keep_history' => '30'},
        'history_str' => { 'period' => 'day', 'keep_history' => '30'},
        'history_text' => { 'period' => 'day', 'keep_history' => '30'},
        'history_uint' => { 'period' => 'day', 'keep_history' => '15'},
        'trends' => { 'period' => 'month', 'keep_history' => '12'},
        'trends_uint' => { 'period' => 'month', 'keep_history' => '12'},

# comment next 5 lines if you partition zabbix database starting from 2.2
# they usually used for zabbix database before 2.2

#       'acknowledges' => { 'period' => 'month', 'keep_history' => '23'},
#       'alerts' => { 'period' => 'month', 'keep_history' => '6'},
#       'auditlog' => { 'period' => 'month', 'keep_history' => '24'},
#       'events' => { 'period' => 'month', 'keep_history' => '12'},
#       'service_alarms' => { 'period' => 'month', 'keep_history' => '6'},
        };
my $amount_partitions = 10;

my $curr_tz = 'Europe/Moscow';

my $part_tables;

my $dbh = DBI->connect($dsn, $db_user_name, $db_password, {'ShowErrorStatement' => 1});

unless ( check_have_partition() ) {
    print "Your installation of MySQL does not support table partitioning.\n";
    syslog(LOG_CRIT, 'Your installation of MySQL does not support table partitioning.');
    exit 1;
}

my $sth = $dbh->prepare(qq{SELECT table_name, partition_name, lower(partition_method) as partition_method,
                    rtrim(ltrim(partition_expression)) as partition_expression,
                    partition_description, table_rows
                FROM information_schema.partitions
                WHERE partition_name IS NOT NULL AND table_schema = ?});
$sth->execute($db_schema);

while (my $row =  $sth->fetchrow_hashref()) {
    $part_tables->{$row->{'table_name'}}->{$row->{'partition_name'}} = $row;
}

$sth->finish();

foreach my $key (sort keys %{$tables}) {
    unless (defined($part_tables->{$key})) {
        syslog(LOG_ERR, 'Partitioning for "'.$key.'" is not found! The table might be not partitioned.');
        next;
    }

    create_next_partition($key, $part_tables->{$key}, $tables->{$key}->{'period'});
    remove_old_partitions($key, $part_tables->{$key}, $tables->{$key}->{'period'}, $tables->{$key}->{'keep_history'})
}

delete_old_data();

$dbh->disconnect();

sub check_have_partition {
    my $result = 0;
# MySQL 5.5
    my $sth = $dbh->prepare(qq{SELECT variable_value FROM information_schema.global_variables WHERE variable_name = 'have_partitioning'});
# MySQL 5.6
    #my $sth = $dbh->prepare(qq{SELECT plugin_status FROM information_schema.plugins WHERE plugin_name = 'partition'});

    $sth->execute();

    my $row = $sth->fetchrow_array();

    $sth->finish();

# MySQL 5.5
    return 1 if $row eq 'YES';
# MySQL 5.6
    #return 1 if $row eq 'ACTIVE';
}

sub create_next_partition {
    my $table_name = shift;
    my $table_part = shift;
    my $period = shift;

    for (my $curr_part = 0; $curr_part < $amount_partitions; $curr_part++) {
        my $next_name = name_next_part($tables->{$table_name}->{'period'}, $curr_part);
        my $found = 0;

        foreach my $partition (sort keys %{$table_part}) {
            if ($next_name eq $partition) {
                syslog(LOG_INFO, "Next partition for $table_name table has already been created. It is $next_name");
                $found = 1;
            }
        }

        if ( $found == 0 ) {
            syslog(LOG_INFO, "Creating a partition for $table_name table ($next_name)");
            my $query = 'ALTER TABLE '."$db_schema.$table_name".' ADD PARTITION (PARTITION '.$next_name.
                        ' VALUES less than (UNIX_TIMESTAMP("'.date_next_part($tables->{$table_name}->{'period'}, $curr_part).'") div 1))';
            syslog(LOG_DEBUG, $query);
            $dbh->do($query);
        }
    }
}

sub remove_old_partitions {
    my $table_name = shift;
    my $table_part = shift;
    my $period = shift;
    my $keep_history = shift;

    my $curr_date = DateTime->now;
    $curr_date->set_time_zone( $curr_tz );

    if ( $period eq 'day' ) {
        $curr_date->add(days => -$keep_history);
        $curr_date->add(hours => -$curr_date->strftime('%H'));
        $curr_date->add(minutes => -$curr_date->strftime('%M'));
        $curr_date->add(seconds => -$curr_date->strftime('%S'));
    }
    elsif ( $period eq 'week' ) {
    }
    elsif ( $period eq 'month' ) {
        $curr_date->add(months => -$keep_history);

        $curr_date->add(days => -$curr_date->strftime('%d')+1);
        $curr_date->add(hours => -$curr_date->strftime('%H'));
        $curr_date->add(minutes => -$curr_date->strftime('%M'));
        $curr_date->add(seconds => -$curr_date->strftime('%S'));
    }

    foreach my $partition (sort keys %{$table_part}) {
        if ($table_part->{$partition}->{'partition_description'} <= $curr_date->epoch) {
            syslog(LOG_INFO, "Removing old $partition partition from $table_name table");

            my $query = "ALTER TABLE $db_schema.$table_name DROP PARTITION $partition";

            syslog(LOG_DEBUG, $query);
            $dbh->do($query);
        }
    }
}

sub name_next_part {
    my $period = shift;
    my $curr_part = shift;

    my $name_template;

    my $curr_date = DateTime->now;
    $curr_date->set_time_zone( $curr_tz );

    if ( $period eq 'day' ) {
        my $curr_date = $curr_date->truncate( to => 'day' );
        $curr_date->add(days => 1 + $curr_part);

        $name_template = $curr_date->strftime('p%Y_%m_%d');
    }
    elsif ($period eq 'week') {
        my $curr_date = $curr_date->truncate( to => 'week' );
        $curr_date->add(days => 7 * $curr_part);

        $name_template = $curr_date->strftime('p%Y_%m_w%W');
    }
    elsif ($period eq 'month') {
        my $curr_date = $curr_date->truncate( to => 'month' );
        $curr_date->add(months => 1 + $curr_part);

        $name_template = $curr_date->strftime('p%Y_%m');
    }

    return $name_template;
}

sub date_next_part {
    my $period = shift;
    my $curr_part = shift;

    my $period_date;

    my $curr_date = DateTime->now;
    $curr_date->set_time_zone( $curr_tz );

    if ( $period eq 'day' ) {
        my $curr_date = $curr_date->truncate( to => 'day' );
        $curr_date->add(days => 2 + $curr_part);
        $period_date = $curr_date->strftime('%Y-%m-%d');
    }
    elsif ($period eq 'week') {
        my $curr_date = $curr_date->truncate( to => 'week' );
        $curr_date->add(days => 7 * $curr_part + 1);
        $period_date = $curr_date->strftime('%Y-%m-%d');
    }
    elsif ($period eq 'month') {
        my $curr_date = $curr_date->truncate( to => 'month' );
        $curr_date->add(months => 2 + $curr_part);

        $period_date = $curr_date->strftime('%Y-%m-%d');
    }

    return $period_date;
}

sub delete_old_data {
    $dbh->do("DELETE FROM sessions WHERE lastaccess < UNIX_TIMESTAMP(NOW() - INTERVAL 1 MONTH)");
    $dbh->do("TRUNCATE housekeeper");
    $dbh->do("DELETE FROM auditlog_details WHERE NOT EXISTS (SELECT NULL FROM auditlog WHERE auditlog.auditid = auditlog_details.auditid)");
}
        
    
Built on GitHub with MkDocs and Pure theme