PostgreSQL La base de donnees la plus sophistiquee au monde.

La planète francophone de PostgreSQL

dimanche 19 avril 2015

Guillaume Lelarge

Analyse du VACUUM

Ce billet fait partie d'une série sur l'écriture de mon livre, « PostgreSQL - Architecture et notions avancées ».

Et voilà, deux nouveaux chapitres écrits, dont un sur le système transactionnel de PostgreSQL. Ce dernier m'a demandé d' étudier plus attentivement le travail de l'opération VACUUM. J'en connaissais le principe et son fonctionnement, à savoir un fonctionnement en trois phases : recherche des éléments à flagguer comme invisibles, suppression de ces éléments dans les index, puis suppression dans la table (pas physiquement). Cependant, je ne l'avais pas regardé plus précisément.

J'ai donc lu le code, puis écrit un petit patch pour mieux suivre cela (disponible en pièce jointe). J'ai exécuté un script SQL pour visualiser différents comportements. Par exemple, on ajoute dix lignes dans une nouvelle table, puis on met à jour une ligne sur trois, et enfin on exécute un VACUUM sur cette table :

CREATE TABLE t2(c1 integer);
ALTER TABLE t2 SET (autovacuum_enabled=off);
INSERT INTO t2 SELECT generate_series(1, 10);
UPDATE t2 SET c1=-c1 where c1%3=1;
SET client_min_messages to log;
VACUUM t2;

Voici le log fourni par le patch :

psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[117] - VACUUM on 0, toast included, no wraparound
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[249] - vacuuming 82231
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum_rel[1207] - relation t2 (82231) opened with lockmode 4
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_rel[194] - vacuuming...
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[465] - relation has 1 blocks
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 4 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 11
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 12
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 7 is now REDIRECTed to item 13
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 10 is now REDIRECTed to item 14
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (10 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 320)

Il n'y a là qu'une seule étape exécutée. En effet, dû au très petit nombre de lignes dans la table, le seul bloc de 8 Ko n'a pas été entièrement occupé. Du coup, PostgreSQL place les nouvelles versions des lignes mises à jour dans le même bloc que les anciennes versions et utilise le Heap Over Tuple pour lier les enregistrements. De plus, comme il n'y a pas d'index, pas besoin de les mettre à jour.

Maintenant, faisons la même chose avec 400 lignes (en fait, suffisamment pour remplir plus d'un bloc). Les logs sont beaucoup plus importants.

psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[117] - VACUUM on 0, toast included, no wraparound
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[249] - vacuuming 82234
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum_rel[1207] - relation t2 (82234) opened with lockmode 4
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_rel[194] - vacuuming...
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[465] - relation has 3 blocks
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 76 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 1 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 4 is now DEAD
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 223 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 226 is now DEAD
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (150 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 4800)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 1 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 4 DEAD
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 223 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 226 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1237] - block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 1 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 4 is now UNUSED
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 223 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 226 is now UNUSED
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (150 have storage, 76 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 4800)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 58 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 3 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 6 is now DEAD
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 171 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 174 is now DEAD
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (168 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5376)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 3 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 6 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 9 DEAD
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 171 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 174 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1237] - block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 3 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 6 is now UNUSED
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 171 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 174 is now UNUSED
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (168 have storage, 58 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5376)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 2
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 0 deleted items found

Chaque élément devenu invisible est déclaré DEAD lors de la première étape, puis UNUSED à la troisième étape.

Il est dit que le fillfactor permet d'augmenter l'utilisation du Heap Over Tuple. Voici ce que cela donne avec un fillfactor à 90% :

psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[117] - VACUUM on 0, toast included, no wraparound
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[249] - vacuuming 82237
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum_rel[1207] - relation t2 (82237) opened with lockmode 4
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_rel[194] - vacuuming...
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[465] - relation has 3 blocks
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 68 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 205
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 206
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 61 is now REDIRECTed to item 225
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 64 is now REDIRECTed to item 226
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 67 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 70 is now DEAD
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 199 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 202 is now DEAD
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (158 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5056)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 67 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 70 DEAD
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 199 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 202 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1237] - block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 67 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 70 is now UNUSED
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 199 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 202 is now UNUSED
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (158 have storage, 46 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5056)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 66 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 205
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 206
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 61 is now REDIRECTed to item 225
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 64 is now REDIRECTed to item 226
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 67 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 70 is now DEAD
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 193 is now DEAD
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[702] - item 196 is now DEAD
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (160 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5120)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 67 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 70 DEAD
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 193 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[817] - item 196 DEAD
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1237] - block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 67 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 70 is now UNUSED
[...]
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 193 is now UNUSED
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_page[1250] - item 196 is now UNUSED
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (160 have storage, 44 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 5120)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 2
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 0 deleted items found

Certains enregistrements bénéficient de HOT, mais la majorité deviennent UNUSED. Essayons avec un fillfactor de 50% :

psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[117] - VACUUM on 0, toast included, no wraparound
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum[249] - vacuuming 82240
psql:script.sql:9: LOG:  patch - vacuum.c - vacuum_rel[1207] - relation t2 (82240) opened with lockmode 4
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_vacuum_rel[194] - vacuuming...
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[465] - relation has 4 blocks
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 0
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 38 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 114
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 115
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 109 is now REDIRECTed to item 150
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 112 is now REDIRECTed to item 151
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (113 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 3616)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 1
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 38 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 2 is now REDIRECTed to item 114
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 5 is now REDIRECTed to item 115
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 110 is now REDIRECTed to item 150
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 113 is now REDIRECTed to item 151
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (113 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 3616)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 2
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 37 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 3 is now REDIRECTed to item 114
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 6 is now REDIRECTed to item 115
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 108 is now REDIRECTed to item 149
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 111 is now REDIRECTed to item 150
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (113 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 3616)
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[534] - working on block 3
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[629] - reading block
psql:script.sql:9: LOG:  patch - vacuumlazy.c - lazy_scan_heap[762] - pruning HOT update chains...
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune[220] - 21 deleted items found
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 1 is now REDIRECTed to item 62
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 4 is now REDIRECTed to item 63
[...]
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 58 is now REDIRECTed to item 81
psql:script.sql:9: LOG:  patch - pruneheap.c - heap_page_prune_execute[691] - item 61 is now REDIRECTed to item 82
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[489] - compacting page (61 have storage, 0 are unused)
psql:script.sql:9: LOG:  patch - bufpage.c - PageRepairFragmentation[514] - compacting page (totallen 1952)

Dans ce cas, tous les enregistrements bénéficient de HOT, aucun n'est UNUSED. Cela permet des mises à jour et une maintenance plus rapides, mais c'est au prix d'une table plus volumineuse (sur disque et dans le cache des relations de PostgreSQL).

Et du coup, vous vous demandez peut-être quand va sortir la version beta 0.3 du livre ? D'ici peu a priori. Un peu de relecture, quelques ajustements de dernières minutes, et ça devrait être prêt :)

par Guillaume Lelarge le dimanche 19 avril 2015 à 17h21

mardi 14 avril 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 12 avril 2015

L'appel à conférenciers pour le PGDay de Belfort (France) se termine le 13 avril 2015. La conférence aura lieu le 2 juin : http://select-2-6-2015-as-pgday.org

La 4ème conférence PostgreSQL turque se tiendra à Istanbul le 9 mai 2015 : http://pgday.postgresql.org.tr/en/index.html

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Fujii Masao a poussé :

Álvaro Herrera a poussé :

  • Fix object identities for pg_conversion objects. This was already fixed in 0d906798f, but I failed to update the array-formatted case. This is not backpatched, since this only affects the code path introduced by commit a676201490c. http://git.postgresql.org/pg/commitdiff/70dc2db7f1dfdecdacf595bf00964cb20ad5a835
  • pg_event_trigger_dropped_objects: add is_temp column. It now also reports temporary objects dropped that are local to the backend. Previously we weren't reporting any temp objects because it was deemed unnecessary; but as it turns out, it is necessary if we want to keep close track of DDL command execution inside one session. Temp objects are reported as living in schema pg_temp, which works because such a schema-qualification always refers to the temp objects of the current session. http://git.postgresql.org/pg/commitdiff/e9a077cad3799b41e8deef6fd8cb87e50164a791
  • Remove variable shadowing. Commit a2e35b53 should have removed the variable declaration in the inner block, but didn't. As a result, the returned address might end up not being what was intended. http://git.postgresql.org/pg/commitdiff/4e17e32f53c2de4a862ee5ef8bdcfa9152c11e25
  • Fix autovacuum launcher shutdown sequence. It was previously possible to have the launcher re-execute its main loop before shutting down if some other signal was received or an error occurred after getting SIGTERM, as reported by Qingqing Zhou. While investigating, Tom Lane further noticed that if autovacuum had been disabled in the config file, it would misbehave by trying to start a new worker instead of bailing out immediately -- it would consider itself as invoked in emergency mode. Fix both problems by checking the shutdown flag in a few more places. These problems have existed since autovacuum was introduced, so backpatch all the way back. http://git.postgresql.org/pg/commitdiff/5df64f298d2863c9fb39437abb3ae6f988aedc0a
  • Change SQLSTATE for event triggers "wrong context" message. When certain event-trigger-only functions are called when not in the wrong context, they were reporting the "feature not supported" SQLSTATE, which is somewhat misleading. Create a new custom error code for such uses instead. Not backpatched since it may be seen as an undesirable behavioral change. Author: Michael Paquier. Discussion: https://www.postgresql.org/message-id/CAB7nPqQ-5NAkHQHh_NOm7FPep37NCiLKwPoJ2Yxb8TDoGgbYYA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/73206812cd97436cffd8f331dbb09d38a2728162
  • Optimize locking a tuple already locked by another subxact. Locking and updating the same tuple repeatedly led to some strange multixacts being created which had several subtransactions of the same parent transaction holding locks of the same strength. However, once a subxact of the current transaction holds a lock of a given strength, it's not necessary to acquire the same lock again. This made some coding patterns much slower than required. The fix is twofold. First we change HeapTupleSatisfiesUpdate to return HeapTupleBeingUpdated for the case where the current transaction is already a single-xid locker for the given tuple; it used to return HeapTupleMayBeUpdated for that case. The new logic is simpler, and the change to pgrowlocks is a testament to that: previously we needed to check for the single-xid locker separately in a very ugly way. That test is simpler now. As fallout from the HTSU change, some of its callers need to be amended so that tuple-locked-by-own-transaction is taken into account in the BeingUpdated case rather than the MayBeUpdated case. For many of them there is no difference; but heap_delete() and heap_update now check explicitely and do not grab tuple lock in that case. The HTSU change also means that routine MultiXactHasRunningRemoteMembers introduced in commit 11ac4c73cb895 is no longer necessary and can be removed; the case that used to require it is now handled naturally as result of the changes to heap_delete and heap_update. The second part of the fix to the performance issue is to adjust heap_lock_tuple to avoid the slowness: 1. Previously we checked for the case that our own transaction already held a strong enough lock and returned MayBeUpdated, but only in the multixact case. Now we do it for the plain Xid case as well, which saves having to LockTuple. 2. If the current transaction is the only locker of the tuple (but with a lock not as strong as what we need; otherwise it would have been caught in the check mentioned above), we can skip sleeping on the multixact, and instead go straight to create an updated multixact with the additional lock strength. 3. Most importantly, make sure that both the single-xid-locker case and the multixact-locker case optimization are applied always. We do this by checking both in a single place, rather than them appearing in two separate portions of the routine -- something that is made possible by the HeapTupleSatisfiesUpdate API change. Previously we would only check for the single-xid case when HTSU returned MayBeUpdated, and only checked for the multixact case when HTSU returned BeingUpdated. This was at odds with what HTSU actually returned in one case: if our own transaction was locker in a multixact, it returned MayBeUpdated, so the optimization never applied. This is what led to the large multixacts in the first place. Per bug report #8470 by Oskari Saarenmaa. http://git.postgresql.org/pg/commitdiff/27846f02c176eebe7e08ce51ed4d52140454e196

Simon Riggs a poussé :

Heikki Linnakangas a poussé :

Tom Lane a poussé :

Robert Haas a poussé :

Bruce Momjian a poussé :

Andres Freund a poussé :

Magnus Hagander a poussé :

Peter Eisentraut a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Fabrízio de Royes Mello sent in two more revisions of a patch to refactor reloptions to set locklevel.
  • Artem Luzyanin sent in two revisions of a patch to consolidate the documentation of spinlocks and like kind items.
  • Emre Hasegeli sent in another revision of a patch to add a BRIN range operator class.
  • Tomas Vondra sent in a patch to use foreign keys to improve join estimates.
  • Michael Paquier sent in a patch to ignore some binaries generated in src/test.
  • SAWADA Masahiko and Fabrízio de Royes Mello traded patches to add REINDEX ... VERBOSE.
  • Shigeru HANADA sent in four more revisions of a patch to add a foreign join API.
  • Petr Jelinek sent in another revision of a patch to implement TABLESAMPLE.
  • Craig Ringer sent in a patch to add a pid column to pg_replication_slots.
  • Peter Eisentraut and Pavel Stehule traded patches to add TRANSFORMS.
  • Michael Paquier sent in another revision of a patch to add an error code to track unsupported contexts.
  • Álvaro Herrera sent in two more revisions of a patch to add deparsing utilities.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT UPDATE (and IGNORE).
  • Tom Lane sent in another revision of a patch to implement UPDATE (*) SET ...
  • Fujii Masao sent in another revision of a patch to remove the obsolete FORCE option from REINDEX.
  • Dean Rasheed sent in another revision of a patch to fix some infelicities in the error reporting for row-level access control.
  • Kyotaro HORIGUCHI sent in another revision of a patch to implement regnamespace and regrole.
  • Antonin Houska sent in a patch to fix some issues in xlogreader.
  • Craig Ringer sent in two revisions of a patch to make pg_dump -t take materialized views, matview data, foreign tables, and sequences.
  • Bruce Momjian sent in two revisions of a patch to ensure that CREATE TABLE (LIKE...) preserves the relhasoids setting.
  • Michael Paquier sent in another revision of a patch to suppport TAP tests with MSVC and Windows.
  • Jan Urbański sent in another revision of a patch to fix a bug in libpq's multi-threaded SSL callback handling.
  • Ian Stakenvicius sent in a patch to fix an issue on Gentoo where postgres fails to start with timezone-data >=2013e.
  • Etsuro Fujita sent in another revision of a patch to fix a problem where EvalPlanQual behaves oddly for FDW queries involving system columns.
  • Pavel Stehule sent in a patch to add a "raw" output option to COPY.
  • Chen Huajun sent in a patch to prevent setting Win32 server-side socket buffer size on Windows 2012.
  • David Rowley sent in a patch to fix a few appendStringInfo* calls that were not quite doing things the way as intended.
  • Michael Paquier sent in another revision of a patch to help improve the performance of make check-world.
  • Heikki Linnakangas sent in a patch to remove xlogrecord padding.

par N Bougain le mardi 14 avril 2015 à 21h05

samedi 11 avril 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 5 avril 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en avril

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Fix multiple bugs and infelicities in pg_rewind. Bugs all spotted by Coverity, including wrong realloc() size request and memory leaks. Cosmetic improvements by me. The usage of the global variable "filemap" here is still pretty awful, but at least I got rid of the gratuitous aliasing in several routines (which was helping to annoy Coverity, as well as being a bug risk). http://git.postgresql.org/pg/commitdiff/c67f366fa9f748257861ee233b47b80eb5ffa857
  • Clean up all the cruft after a pg_rewind test run. regress_log temp directory was properly .gitignore'd, which may explain why it got left out of the "make clean" action. http://git.postgresql.org/pg/commitdiff/1c41e2a998a0de16d9d33949a7b98a5be3d2477c
  • Fix rare core dump in BackendIdGetTransactionIds(). BackendIdGetTransactionIds() neglected the possibility that the PROC pointer in a ProcState array entry is null. In current usage, this could only crash if the other backend had exited since pgstat_read_current_status saw it as active, which is a pretty narrow window. But it's reachable in the field, per bug #12918 from Vladimir Borodin. Back-patch to 9.4 where the faulty code was introduced. http://git.postgresql.org/pg/commitdiff/701dcc983eb4d08dd36bb3a0ddba255819797760
  • Be more careful about printing constants in ruleutils.c. The previous coding in get_const_expr() tried to avoid quoting integer, float, and numeric literals if at all possible. While that looks nice, it means that dumped expressions might re-parse to something that's semantically equivalent but not the exact same parsetree; for example a FLOAT8 constant would re-parse as a NUMERIC constant with a cast to FLOAT8. Though the result would be the same after constant-folding, this is problematic in certain contexts. In particular, Jeff Davis pointed out that this could cause unexpected failures in ALTER INHERIT operations because of child tables having not-exactly-equivalent CHECK expressions. Therefore, favor correctness over legibility and dump such constants in quotes except in the limited cases where they'll be interpreted as the same type even without any casting. This results in assorted small changes in the regression test outputs, and will affect display of user-defined views and rules similarly. The odds of that causing problems in the field seem non-negligible; given the lack of previous complaints, it seems best not to change this in the back branches. http://git.postgresql.org/pg/commitdiff/542320c2bd0b3796a8a9a4617cdb23fbad473390
  • Fix bogus concurrent use of _hash_getnewbuf() in bucket split code. _hash_splitbucket() obtained the base page of the new bucket by calling _hash_getnewbuf(), but it held no exclusive lock that would prevent some other process from calling _hash_getnewbuf() at the same time. This is contrary to _hash_getnewbuf()'s API spec and could in fact cause failures. In practice, we must only call that function while holding write lock on the hash index's metapage. An additional problem was that we'd already modified the metapage's bucket mapping data, meaning that failure to extend the index would leave us with a corrupt index. Fix both issues by moving the _hash_getnewbuf() call to just before we modify the metapage in _hash_expandtable(). Unfortunately there's still a large problem here, which is that we could also incur ENOSPC while trying to get an overflow page for the new bucket. That would leave the index corrupt in a more subtle way, namely that some index tuples that should be in the new bucket might still be in the old one. Fixing that seems substantially more difficult; even preallocating as many pages as we could possibly need wouldn't entirely guarantee that the bucket split would complete successfully. So for today let's just deal with the base case. Per report from Antonin Houska. Back-patch to all active branches. http://git.postgresql.org/pg/commitdiff/ed9cc2b5df59fdbc50cce37399e26b03ab2c1686
  • Fix incorrect markup in documentation of window frame clauses. You're required to write either RANGE or ROWS to start a frame clause, but the documentation incorrectly implied this is optional. Noted by David Johnston. http://git.postgresql.org/pg/commitdiff/f6caf5acf1def92d7425151a92fd990c566fdcc3
  • Provide real selectivity estimators for inet/cidr operators. This patch fills in the formerly-stub networksel() and networkjoinsel() estimation functions. Those are used for << <<= >> >>= and && operators on inet/cidr types. The estimation is not perfect, certainly, because we rely on the existing statistics collected for the inet btree operators. But it's a long way better than nothing, and it's not clear that asking ANALYZE to collect separate stats for these operators would be a win. Emre Hasegeli, with reviews from Dilip Kumar and Heikki Linnakangas, and some further hacking by me http://git.postgresql.org/pg/commitdiff/89840d7d3fa943cb932f6a00707fdb17a9cab001
  • Fix rare startup failure induced by MVCC-catalog-scans patch. While a new backend nominally participates in sinval signaling starting from the SharedInvalBackendInit call near the top of InitPostgres, it cannot recognize sinval messages for unshared catalogs of its database until it has set up MyDatabaseId. This is not problematic for the catcache or relcache, which by definition won't have loaded any data from or about such catalogs before that point. However, commit 568d4138c646cd7c introduced a mechanism for re-using MVCC snapshots for catalog scans, and made invalidation of those depend on recognizing relevant sinval messages. So it's possible to establish a catalog snapshot to read pg_authid and pg_database, then before we set MyDatabaseId, receive sinval messages that should result in invalidating that snapshot --- but do not, because we don't realize they are for our database. This mechanism explains the intermittent buildfarm failures we've seen since commit 31eae6028eca4365. That commit was not itself at fault, but it introduced a new regression test that does reconnections concurrently with the "vacuum full pg_am" command in vacuum.sql. This allowed the pre-existing error to be exposed, given just the right timing, because we'd fail to update our information about how to access pg_am. In principle any VACUUM FULL on a system catalog could have created a similar hazard for concurrent incoming connections. Perhaps there are more subtle failure cases as well. To fix, force invalidation of the catalog snapshot as soon as we've set MyDatabaseId. Back-patch to 9.4 where the error was introduced. http://git.postgresql.org/pg/commitdiff/bc49d9324a464fce8f60e1bc14531631883021d4
  • Remove unnecessary variables in _hash_splitbucket(). Commit ed9cc2b5df59fdbc50cce37399e26b03ab2c1686 made it unnecessary to pass start_nblkno to _hash_splitbucket(), and for that matter unnecessary to have the internal nblkno variable either. My compiler didn't complain about that, but some did. I also rearranged the use of oblkno a bit to make that case more parallel. Report and initial patch by Petr Jelinek, rearranged a bit by me. Back-patch to all branches, like the previous patch. http://git.postgresql.org/pg/commitdiff/b7e1652d5de8b618c0204588969c8b59d12e9361
  • Fix TAP tests to use only standard command-line argument ordering. Some of the TAP tests were supposing that PG programs would accept switches after non-switch arguments on their command lines. While GNU getopt_long() does allow that, our own implementation does not, and it's nowhere suggested in our documentation that such cases should work. Adjust the tests to use only the documented syntax. Back-patch to 9.4, since without this the TAP tests fail when run with src/port's getopt_long() implementation. Michael Paquier http://git.postgresql.org/pg/commitdiff/c67a86f7da90c30b81f91957023fb752f06f0598
  • Fix incorrect matching of subexpressions in outer-join plan nodes. Previously we would re-use input subexpressions in all expression trees attached to a Join plan node. However, if it's an outer join and the subexpression appears in the nullable-side input, this is potentially incorrect for apparently-matching subexpressions that came from above the outer join (ie, targetlist and qpqual expressions), because the executor will treat the subexpression value as NULL when maybe it should not be. The case is fairly hard to hit because (a) you need a non-strict subexpression (else NULL is correct), and (b) we don't usually compute expressions in the outputs of non-toplevel plan nodes. But we might do so if the expressions are sort keys for a mergejoin, for example. Probably in the long run we should make a more explicit distinction between Vars appearing above and below an outer join, but that will be a major planner redesign and not at all back-patchable. For the moment, just hack set_join_references so that it will not match any non-Var expressions coming from nullable inputs to expressions that came from above the join. (This is somewhat overkill, in that a strict expression could still be matched, but it doesn't seem worth the effort to check that.) Per report from Qingqing Zhou. The added regression test case is based on his example. This has been broken for a very long time, so back-patch to all active branches. http://git.postgresql.org/pg/commitdiff/ca6805338fba010cc3f8b842905d7a62e280b7ab
  • Suppress clang's unhelpful gripes about -pthread switch being unused. Considering the number of cases in which "unused" command line arguments are silently ignored by compilers, it's fairly astonishing that anybody thought this warning was useful; it's certainly nothing but an annoyance when building Postgres. One such case is that neither gcc nor clang complain about unrecognized -Wno-foo switches, making it more difficult to figure out whether the switch does anything than one could wish. Back-patch to 9.3, which is as far back as the patch applies conveniently (we'd have to back-patch PGAC_PROG_CC_VAR_OPT to go further, and it doesn't seem worth that). http://git.postgresql.org/pg/commitdiff/73b416b2e41237b657d29d8f42a4bb34bf700928

Heikki Linnakangas a poussé :

Álvaro Herrera a poussé :

  • Fix lost persistence setting during REINDEX INDEX. ReindexIndex() trusts a parser-built RangeVar with the persistence to use for the new copy of the index; but the parser naturally does not know what's the persistence of the original index. To find out the correct persistence, grab it from relcache. This bug was introduced by commit 85b506bbfc2937c9, and therefore no backpatch is necessary. Bug reported by Thom Brown, analysis and patch by Michael Paquier; test case provided by Fabrízio de Royes Mello. http://git.postgresql.org/pg/commitdiff/0853630159944bb3652336602ff5f7f62cd27a5a
  • Change array_offset to return subscripts, not offsets ... and rename it and its sibling array_offsets to array_position and array_positions, to account for the changed behavior. Having the functions return subscripts better matches existing practice, and is better suited to using the result value as a subscript into the array directly. For one-based arrays, the new definition is identical to what was originally committed. (We use the term "subscript" in the documentation, which is what we use whenever we talk about arrays; but the functions themselves are named using the word "position" to match the standard-defined POSITION() functions.) Author: Pavel Stěhule. Behavioral problem noted by Dean Rasheed. http://git.postgresql.org/pg/commitdiff/97690ea6e86c412461dd5dc99953b829564d1a55
  • psql: fix \connect with URIs and conninfo strings. psql was already accepting conninfo strings as the first parameter in \connect, but the way it worked wasn't sane; some of the other parameters would get the previous connection's values, causing it to connect to a completely unexpected server or, more likely, not finding any server at all because of completely wrong combinations of parameters. Fix by explicitely checking for a conninfo-looking parameter in the dbname position; if one is found, use its complete specification rather than mix with the other arguments. Also, change tab-completion to not try to complete conninfo/URI-looking "dbnames" and document that conninfos are accepted as first argument. There was a weak consensus to backpatch this, because while the behavior of using the dbname as a conninfo is nowhere documented for \connect, it is reasonable to expect that it works because it does work in many other contexts. Therefore this is backpatched all the way back to 9.0. To implement this, routines previously private to libpq have been duplicated so that psql can decide what looks like a conninfo/URI string. In back branches, just duplicate the same code all the way back to 9.2, where URIs where introduced; 9.0 and 9.1 have a simpler version. In master, the routines are moved to src/common and renamed. Author: David Fetter, Andrew Dunstan. Some editorialization by me (probably earning a Gierth's "Sloppy" badge in the process.) Reviewers: Andrew Gierth, Erik Rijkers, Pavel Stěhule, Stephen Frost, Robert Haas, Andrew Dunstan. http://git.postgresql.org/pg/commitdiff/fcef1617295c074f2684c887627184d2fc26ac04
  • psql: fix \connect with URIs and conninfo strings. This is the second try at this, after fcef1617295 failed miserably and had to be reverted: as it turns out, libpq cannot depend on libpgcommon after all. Instead of shuffling code in the master branch, make that one just like 9.4 and accept the duplication. (This was all my own mistake, not the patch submitter's). psql was already accepting conninfo strings as the first parameter in \connect, but the way it worked wasn't sane; some of the other parameters would get the previous connection's values, causing it to connect to a completely unexpected server or, more likely, not finding any server at all because of completely wrong combinations of parameters. Fix by explicitely checking for a conninfo-looking parameter in the dbname position; if one is found, use its complete specification rather than mix with the other arguments. Also, change tab-completion to not try to complete conninfo/URI-looking "dbnames" and document that conninfos are accepted as first argument. There was a weak consensus to backpatch this, because while the behavior of using the dbname as a conninfo is nowhere documented for \connect, it is reasonable to expect that it works because it does work in many other contexts. Therefore this is backpatched all the way back to 9.0. Author: David Fetter, Andrew Dunstan. Some editorialization by me (probably earning a Gierth's "Sloppy" badge in the process.) Reviewers: Andrew Gierth, Erik Rijkers, Pavel Stěhule, Stephen Frost, Robert Haas, Andrew Dunstan. http://git.postgresql.org/pg/commitdiff/e146ca682062ca1f5015f3820571c5359f5f9dba
  • autovacuum: Fix polarity of "wraparound" variable. Commit 0d831389749a3 inadvertently reversed the meaning of the wraparound variable. This causes vacuums which are not required for wraparound to wait for locks to be acquired, and what is worse, it allows wraparound vacuums to skip locked pages. Bug reported by Jeff Janes in http://www.postgresql.org/message-id/CAMkU=1xmTEiaY=5oMHsSQo5vd9V1Ze4kNLL0qN2eH0P_GXOaYw@mail.gmail.com Analysis and patch by Kyotaro HORIGUCHI http://git.postgresql.org/pg/commitdiff/00ee6c7672fe0bf9448bc744b5e3408f5ebffc2e
  • Have autovacuum workers listen to SIGHUP, too. They have historically ignored it, but it's been said to be useful at times to change their settings mid-flight. Author: Michael Paquier http://git.postgresql.org/pg/commitdiff/a75fb9b335db0e063ece283ebd207530abe1b53b
  • Add log_min_autovacuum_duration per-table option. This is useful to control autovacuum log volume, for situations where monitoring only a set of tables is necessary. Author: Michael Paquier Reviewed by: A team led by Naoya Anzai (also including Akira Kurosawa, Taiki Kondo, Huong Dangminh), Fujii Masao. http://git.postgresql.org/pg/commitdiff/4ff695b17d32a9c330952192dbc789d31a5e2f5e
  • Transform ALTER TABLE/SET TYPE/USING expr during parse analysis. This lets later stages have access to the transformed expression; in particular it allows DDL-deparsing code during event triggers to pass the transformed expression to ruleutils.c, so that the complete command can be deparsed. This shuffles the timing of the transform calls a bit: previously, nothing was transformed during parse analysis, and only the RELKIND_RELATION case was being handled during execution. After this patch, all expressions are transformed during parse analysis (including those for relkinds other than RELATION), and the error for other relation kinds is thrown only during execution. So we do more work than before to reject some bogus cases. That seems acceptable. http://git.postgresql.org/pg/commitdiff/9550e8348b7965715789089555bb5a3fda8c269c

Andrew Dunstan a poussé :

  • Run pg_upgrade and pg_resetxlog with restricted token on Windows. As with initdb these programs need to run with a restricted token, and if they don't pg_upgrade will fail when run as a user with Adminstrator privileges. Backpatch to all live branches. On the development branch the code is reorganized so that the restricted token code is now in a single location. On the stable bramches a less invasive change is made by simply copying the relevant code to pg_upgrade.c and pg_resetxlog.c. Patches and bug report from Muhammad Asif Naeem, reviewed by Michael Paquier, slightly edited by me. http://git.postgresql.org/pg/commitdiff/fa1e5afa8a26d467aec7c8b36a0b749b690f636c
  • Enable float8-byval as the default for 64 bit MSVC builds. This is a long-standing inconsistency that was probably just missed when we got 64 bit MSVC builds. This brings the platform into line with all other systems. http://git.postgresql.org/pg/commitdiff/cf376a4adc0805b0960a5f8e8325fae7d4456926

Bruce Momjian a poussé :

Fujii Masao a poussé :

Simon Riggs a poussé :

Robert Haas a poussé :

Andres Freund a poussé :

  • Define integer limits independently from the system definitions. In 83ff1618 we defined integer limits iff they're not provided by the system. That turns out not to be the greatest idea because there's different ways some datatypes can be represented. E.g. on OSX PG's 64bit datatype will be a 'long int', but OSX unconditionally uses 'long long'. That disparity then can lead to warnings, e.g. around printf formats. One way to fix that would be to back int64 using stdint.h's int64_t. While a good idea it's not that easy to implement. We would e.g. need to include stdint.h in our external headers, which we don't today. Also computing the correct int64 printf formats in that case is nontrivial. Instead simply prefix the integer limits with PG_ and define them unconditionally. I've adjusted all the references to them in code, but not the ones in comments; the latter seems unnecessary to me. Discussion: 20150331141423.GK4878@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/62e2a8dc2c7f6b1351a0385491933af969ed4265

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Michael Paquier sent in a doc patch to describe more precisely the rounding behavior of numeric and double precision.
  • Tomas Vondra sent in another revision of a patch to implement multivariate statistics, useful for among other things cross-column correlations.
  • Heikki Linnakangas sent in a patch to implement SCRAM authentication.
  • David Fetter sent in two more revisions of a patch to allow make_date() to use negative years for BCE dates.
  • Craig Ringer sent in a patch to add views, materialized views, and foreign tables to the meaning of pg_dump's -t command line argument.
  • Fabrízio de Royes Mello sent in a WIP patch to reduce lock level when setting autovacuum reloptions in "ALTER TABLE .. SET ( .. )" statement.
  • Pavel Stehule sent in another revision of a patch to add a row_to_array() function.
  • Michael Paquier sent in two revisions of a patch to add some tests for pg_rewind.
  • Abhijit Menon-Sen sent in two more revisions of a patch to add pgstatbloat to pgstattuple.
  • Haribabu Kommi sent in another revision of a patch to add a catalog view to for pg_hba.conf.
  • Tomas Vondra sent in another revision of a patch to improve the n-distinct estimator.
  • Jehan-Guillaume de Rorthais sent in a patch to document the maximum number of files in the pg_xlog directory.
  • Stephen Frost sent in another revision of a patch to implement role attributes.
  • Bruce Momjian sent in two more revisions of a patch to fix some infelicities in zero-padding in to_char().
  • David Steele sent in another revision of a patch to implement pg_audit.
  • Bruce Momjian sent in two revisions of a patch to fix an infelicity in the SSPI error reporting code.
  • Heikki Linnakangas sent in another revision of a patch to use Intel SSE4.2 CRC instructions where available.
  • Petr Jelinek sent in a patch to remove an unused variable in src/backend/access/hash/hashpage.c.
  • Petr Jelinek sent in another revision of a patch to implement TABLESAMPLE.
  • SAWADA Masahiko sent in a WIP patch to allow making read-only tables.
  • Shigeru HANADA sent in another revision of a patch to add join push-down support to postgres_fdw.
  • Etsuro Fujita sent in another revision of a patch to fix some infelicities in the way EvalPlanQual behaves for FDW queries involving system columns.
  • SAWADA Masahiko sent in another revision of a patch to create a pg_file_settings view.
  • Tomas Vondra sent in a patch to fix to get rid of the spurious warnings caused by the recent change to _hash_splitbucket().
  • Peter Geoghegan sent in a patch to make trace_sort control abbreviation debug output for the text opclass, making it consistent with the numeric opclass.

par N Bougain le samedi 11 avril 2015 à 16h46

Nouvelles hebdomadaires de PostgreSQL - 29 mars 2015

Le pgDay Paris 2015 aura lieu le 21 avril à Paris au siège de La Poste. Les inscriptions sont ouvertes : http://pgday.paris/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mars

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Robert Haas a poussé :

Andres Freund a poussé :

  • Fix copy & paste error in 4f1b890b137. Due to the bug delayed standbys would not delay when applying prepared transactions. Discussion: CAB7nPqT6BO1cCn+sAyDByBxA4EKZNAiPi2mFJ=ANeZmnmewRyg@mail.gmail.com Michael Paquier via Coverity. http://git.postgresql.org/pg/commitdiff/a1105c3dd44c1fb76eb62a708f0421f21b9dde9b
  • Don't delay replication for less than recovery_min_apply_delay's resolution. Recovery delays are implemented by waiting on a latch, and latches take milliseconds as a parameter. The required amount of waiting was computed using microsecond resolution though and the wait loop's abort condition was checking the delay in microseconds as well. This could lead to short spurts of busy looping when the overall wait time was below a millisecond, but above 0 microseconds. Instead just formulate the wait loop's abort condition in millisecond granularity as well. Given that that's recovery_min_apply_delay resolution, it seems harmless to not wait for less than a millisecond. Backpatch to 9.4 where recovery_min_apply_delay was introduced. Discussion: 20150323141819.GH26995@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/87cec51d3ad1107f6f224ed7d773e70c8896e4c0
  • Centralize definition of integer limits. Several submitted and even committed patches have run into the problem that C89, our baseline, does not provide minimum/maximum values for various integer datatypes. C99's stdint.h does, but we can't rely on it. Several parts of the code defined limits locally, so instead centralize the definitions to c.h. This patch also changes the more obvious usages of literal limit values; there's more places that could be changed, but it's less clear whether it's beneficial to change those. Author: Andrew Gierth Discussion: 87619tc5wc.fsf@news-spur.riddles.org.uk http://git.postgresql.org/pg/commitdiff/83ff1618bc9d4e530d3ef2a668a71326784a753c

Heikki Linnakangas a poussé :

Álvaro Herrera a poussé :

  • vacuumdb: Check result status of PQsendQuery. Noticed by Coverity http://git.postgresql.org/pg/commitdiff/871293fb7fa869fbe0f202bef2e0d781bfd21b3e
  • Fix gram.y comment to match reality. There are other comments in there that don't precisely match what's implemented, but this one confused me enough to be worth fixing. http://git.postgresql.org/pg/commitdiff/dc8e05295ab126bc4c943cab3e8e117489ecb246
  • Fix bug for array-formatted identities of user mappings. I failed to realize that server names reported in the object args array would get quoted, which is wrong; remove that, making sure that it's only quoted in the string-formatted identity. This bug was introduced by my commit cf34e373, which was backpatched, but since object name/args arrays are new in commit a676201490c8, there is no need to backpatch this any further. http://git.postgresql.org/pg/commitdiff/b3196e65f5bfc997ec7fa3f91645a09289c10dee
  • Add OID output argument to DefineTSConfiguration ... which is set to the OID of a copied text search config, whenever the COPY clause is used. This is in the spirit of commit a2e35b53c39. http://git.postgresql.org/pg/commitdiff/8217fb1441ce4b4e1785f9acfa0ce50039247a10
  • Return ObjectAddress in many ALTER TABLE sub-routines. Since commit a2e35b53c39b2a, most CREATE and ALTER commands return the ObjectAddress of the affected object. This is useful for event triggers to try to figure out exactly what happened. This patch extends this idea a bit further to cover ALTER TABLE as well: an auxiliary ObjectAddress is returned for each of several subcommands of ALTER TABLE. This makes it possible to decode with precision what happened during execution of any ALTER TABLE command; for instance, which constraint was added by ALTER TABLE ADD CONSTRAINT, or which parent got dropped from the parents list by ALTER TABLE NO INHERIT. As with the previous patch, there is no immediate user-visible change here. This is all really just continuing what c504513f83a9ee8 started. Reviewed by Stephen Frost. http://git.postgresql.org/pg/commitdiff/bdc3d7fa2376a7a1e977383cc3221cfe44c4a893

Tom Lane a poussé :

  • Apply table and domain CHECK constraints in name order. Previously, CHECK constraints of the same scope were checked in whatever order they happened to be read from pg_constraint. (Usually, but not reliably, this would be creation order for domain constraints and reverse creation order for table constraints, because of differing implementation details.) Nondeterministic results of this sort are problematic at least for testing purposes, and in discussion it was agreed to be a violation of the principle of least astonishment. Therefore, borrow the principle already established for triggers, and apply such checks in name order (using strcmp() sort rules). This lets users control the check order if they have a mind to. Domain CHECK constraints still follow the rule of checking lower nested domains' constraints first; the name sort only applies to multiple constraints attached to the same domain. In passing, I failed to resist the temptation to wordsmith a bit in create_domain.sgml. Apply to HEAD only, since this could result in a behavioral change in existing applications, and the potential regression test failures have not actually been observed in our buildfarm. http://git.postgresql.org/pg/commitdiff/e5f455f59fed0632371cddacddd79895b148dc07
  • Fix ExecOpenScanRelation to take a lock on a ROW_MARK_COPY relation. ExecOpenScanRelation assumed that any relation listed in the ExecRowMark list has been locked by InitPlan; but this is not true if the rel's markType is ROW_MARK_COPY, which is possible if it's a foreign table. In most (possibly all) cases, failure to acquire a lock here isn't really problematic because the parser, planner, or plancache would have taken the appropriate lock already. In principle though it might leave us vulnerable to working with a relation that we hold no lock on, and in any case if the executor isn't depending on previously-taken locks otherwise then it should not do so for ROW_MARK_COPY relations. Noted by Etsuro Fujita. Back-patch to all active versions, since the inconsistency has been there a long time. (It's almost certainly irrelevant in 9.0, since that predates foreign tables, but the code's still wrong on its own terms.) http://git.postgresql.org/pg/commitdiff/feeb526cfe33657b8aa8b0cdd45b2ef0d9898877
  • Upgrade src/port/rint.c to be POSIX-compliant. The POSIX spec says that rint() rounds halfway cases to nearest even. Our substitute implementation failed to do that, rather rounding halfway cases away from zero; and it also got some other cases (such as minus zero) wrong. This led to observable cross-platform differences, as reported in bug #12885 from Rich Schaaf; in particular, casting from float to int didn't honor round-to-nearest-even on builds using rint.c. Implement something that attempts to cover all cases per spec, and add some simple regression tests so that we'll notice if any platforms still get this wrong. Although this is a bug fix, no back-patch, as a behavioral change in the back branches was agreed not to be a good idea. Pedro Gimeno Fortea, reviewed by Michael Paquier and myself http://git.postgresql.org/pg/commitdiff/06bf0dd6e354c765403d1331cc9896b360754521
  • Add an ASSERT statement in plpgsql. This is meant to make it easier to insert simple debugging cross-checks in plpgsql functions. Pavel Stehule, reviewed by Jim Nasby http://git.postgresql.org/pg/commitdiff/a4847fc3ef139ba9a8ffebb6ffa06ee72078ffa2
  • Suppress some unused-variable complaints in new LOCK_DEBUG code. Jeff Janes http://git.postgresql.org/pg/commitdiff/bed756a820a2c1ee359f5f5b44806e3599190e95
  • Tweak __attribute__-wrapping macros for better pgindent results. This improves on commit bbfd7edae5aa5ad5553d3c7e102f2e450d4380d4 by making two simple changes: * pg_attribute_noreturn now takes parentheses, ie pg_attribute_noreturn(). Likewise pg_attribute_unused(), pg_attribute_packed(). This reduces pgindent's tendency to misformat declarations involving them. * attributes are now always attached to function declarations, not definitions. Previously some places were taking creative shortcuts, which were not merely candidates for bad misformatting by pgindent but often were outright wrong anyway. (It does little good to put a noreturn annotation where callers can't see it.) In any case, if we would like to believe that these macros can be used with non-gcc compilers, we should avoid gratuitous variance in usage patterns. I also went through and manually improved the formatting of a lot of declarations, and got rid of excessively repetitive (and now obsolete anyway) comments informing the reader what pg_attribute_printf is for. http://git.postgresql.org/pg/commitdiff/785941cdc359c6e595201ffb0df9d28f3f7173a4
  • Better fix for misuse of Float8GetDatumFast(). We can use that macro as long as we put the value into a local variable. Commit 735cd6128 was not wrong on its own terms, but I think this way looks nicer, and it should save a few cycles on 32-bit machines. http://git.postgresql.org/pg/commitdiff/2c33e0fbceb01d0ecd78330feef1315682c64bc4
  • Minor code cleanups in pgbench expression support. Get rid of unnecessary expr_yylex declaration (we haven't supported flex 2.5.4 in a long time, and even if we still did, the declaration in pgbench.h makes this one unnecessary and inappropriate). Fix copyright dates, improve some layout choices, etc. http://git.postgresql.org/pg/commitdiff/e9dd03c03aeed6486129e3101695b13d469c2985
  • Remove a couple other vestigial yylex() declarations. These were workarounds for a long-gone flex bug; all supported versions of flex emit an extern declaration as expected. http://git.postgresql.org/pg/commitdiff/9a8e23311cac14168df6644e03d533a4b07f933e
  • Make ginbuild's funcCtx be independent of its tmpCtx. Previously the funcCtx was a child of the tmpCtx, but that was broken by commit eaa5808e8ec4e82ce1a87103a6b6f687666e4e4c, which made MemoryContextReset() delete, not reset, child contexts. The behavior of having a tmpCtx reset also clear the other context seems rather dubious anyway, so let's just disentangle them. Per report from Erik Rijkers. In passing, fix badly-inaccurate comments about these contexts. http://git.postgresql.org/pg/commitdiff/1601830ec20d56dc7bf6b60a34f69841429e4825
  • Add vacuum_delay_point call in compute_index_stats's per-sample-row loop. Slow functions in index expressions might cause this loop to take long enough to make it worth being cancellable. Probably it would be enough to call CHECK_FOR_INTERRUPTS here, but for consistency with other per-sample-row loops in this file, let's use vacuum_delay_point. Report and patch by Jeff Janes. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/e4cbfd673d530a3e841db26a74f22e11a991205a

Bruce Momjian a poussé :

Kevin Grittner a poussé :

  • Reduce pinning and buffer content locking for btree scans. Even though the main benefit of the Lehman and Yao algorithm for btrees is that no locks need be held between page reads in an index search, we were holding a buffer pin on each leaf page after it was read until we were ready to read the next one. The reason was so that we could treat this as a weak lock to create an "interlock" with vacuum's deletion of heap line pointers, even though our README file pointed out that this was not necessary for a scan using an MVCC snapshot. The main goal of this patch is to reduce the blocking of vacuum processes by in-progress btree index scans (including a cursor which is idle), but the code rearrangement also allows for one less buffer content lock to be taken when a forward scan steps from one page to the next, which results in a small but consistent performance improvement in many workloads. This patch leaves behavior unchanged for some cases, which can be addressed separately so that each case can be evaluated on its own merits. These unchanged cases are when a scan uses a non-MVCC snapshot, an index-only scan, and a scan of a btree index for which modifications are not WAL-logged. If later patches allow all of these cases to drop the buffer pin after reading a leaf page, then the btree vacuum process can be simplified; it will no longer need the "super-exclusive" lock to delete tuples from a page. Reviewed by Heikki Linnakangas and Kyotaro Horiguchi http://git.postgresql.org/pg/commitdiff/2ed5b87f96d473962ec5230fd820abfeaccb2069

Tatsuo Ishii a poussé :

Andrew Dunstan a poussé :

Peter Eisentraut a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Pavel Stehule sent in two more revisions of a patch to add a --strict option to pg_dump for missing tables.
  • Bruce Momjian sent in three more revisions of a patch to add the asciidoc output format to psql.
  • David Steele sent in two more revisions of a patch to to add a pg_audit extension.
  • Peter Geoghegan and Andrew (RhodiumToad) Gierth sent in two more revisions of a patch to add abbreviated keys for NUMERIC.
  • Michael Paquier sent in two revisions of a patch to expose PG_VERSION_NUM to developers.
  • Michael Paquier sent in another revision of a patch to add a table-level log_autovacuum_min_duration.
  • Kyotaro HORIGUCHI sent in a patch to remove individual phases to mark unique joins.
  • Andres Freund sent in another revision of a patch to add replication identifiers.
  • Amit Kapila sent in a patch to fix a bug in dsm_attach().
  • Peter Geoghegan sent in a patch to add DatumGetUInt32() around the hash_any() and hash_uint32() calls within varlena.c.
  • Heikki Linnakangas sent in another revision of a patch to use Intel SSE4.2 CRC instructions where available.
  • Álvaro Herrera sent in another revision of a patch to implement parsing utility commands.
  • Kaigai Kouhei sent in another revision of a patch to add a custom JOIN API.
  • Fabrízio de Royes Mello sent in a patch to fix a bug with indexes on unlogged tables.
  • Amit Kapila sent in two more revisions of a patch to implement parallel sequential scan.
  • Antonin Houska sent in two revisions of a WIP patch to enable spliting hash index buckets.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT IGNORE (and UPDATE).
  • Haribabu Kommi sent in another revision of a patch to add a catalog view to pg_hba.conf.
  • David Rowley sent in another revision of a patch to improve performance for outer joins where the outer side is unique.
  • Tom Lane sent in another revision of a patch to manipulate complex types as non-contiguous structures in-memory.
  • Fabien COELHO sent in another revision of a patch to improve pgbench syntax error messages.
  • Fabien COELHO sent in another revision of a patch to extend pgbench expressions with functions.
  • Pavel Stehule sent in another revision of a patch to add a row_to_array function.
  • Michael Paquier sent in a patch to describe precise rounding behavior of numeric and double precision in the docs.
  • Andres Freund sent in a patch to make heap extension scale better.

par N Bougain le samedi 11 avril 2015 à 16h27

jeudi 2 avril 2015

Dimitri Fontaine

pgDay Paris

Le 21 avril prochain se tient le premier pgDay Paris: une conférence PostgreSQL d'une journée complète. Il s'agit de 8 conférences sur votre base de données préférée par des conférencers internationaux, incluant des retours d'expérience et une analyse de l'utilisation des derniers développements en cours dans notre projet de base de données préféré.

pgDay Paris, 21 avril 2015 : 8 conférences PostgreSQL

Pour 65 € l'accès à la conférence inclut les pauses cafés (avec thé, eau, biscuits, etc) et le déjeuner sur place (repas complet servi à table).

Demandez le programme !

Le programme de pgDay.paris est publié et sa lecture permet de positionner pgDay Paris comme une conférence PostgreSQL incontournable en France.

Nous commencerons avec une présentation de l'utilisation de PostgreSQL dans le groupe ADEO, dont vous connaissez certainement plusieurs des enseignes : les magazins Leroy Merlin, Brico Center ou encore weldom font partie du groupe. Leurs logiciels de caisse utilisent PostgreSQL afin de ne jamais perdre une transaction de paiement !

Ensuite Stéphane Schildknecht nous détaillera un cas d'usage intéressant de PostgreSQL avec de la réplication Slony. Puis Magnus Hagander, committer PostgreSQL et membre de la Core Team du projet, viendra nous présenter les nouveautés de PostgreSQL 9.5, qui va bientôt entrer en période de béta.

Le projet Open Source ToroDB nous sera ensuite présenté par son auteur principal, Álvaro Hernandez. Il s'agit d'une implémentation complète du protocole MongoDB qui utilise PostgreSQL afin de gérer ses données. Pour tout ceux qui se posent la question du NoSQL à comparer au relationnel, ToroDB apporte une réponse intéressante tant du point de vue de l'utilisation en tant que développeur que des performances (volumes stockés, temps d'insertion ou manipulation des données, temps d'exécution des requêtes).

Ensuite Grégoire Hubert viendra lui-même présenter POMM, une approche moderne d'intégration de la base de données et du code orienté objet. Cette approche permet de dépasser les limitations classiques des ORMs et d'utiliser l'ensemble des fonctionnalités du moteur de base de données utilisé !

Nous parlerons égalemment des nouveaux projets de réplication intégrés à PostgreSQL, BDR pour Bi-Directional Replication. Développeur au coeur du projet, Petr Jelinek interviendra afin de montrer comment les solutions BDR et UDR peuvent être utilisées pour résoudre des problématiques classiques de Haute Disponibilité des services et des données avec PostgreSQL.

Les Systèmes d'information géographique seront à l'honneur avec Vincent Picavet, contributeur à PostGIS, qui viendra nous expliquer ce que l'on peut faire de PostgreSQL avec les extension GIS.

Nous terminerons cette journée riche en conférences avec une présentation des progrès de PostgreSQL dans le passé et à l'avenir, ainsi qu'une analyse du positionnement de notre Système de Gestion de Données Relationnelles préféré. Simon Riggs contribue à PostgreSQL depuis plus de 10 ans et a développé des fonctionnalités essentielles : Point In Time Recovery, Hot Standy, Synchronous Replication sont quelques exemples. Il dirige l'effort de développement de l'avenir de la réplication dans PostgreSQL avec Logical Decoding en 9.4 et Bi-Directional Replication. Committer PostgreSQL très actif, DBA certifié Oracle® et Teradata, Simon est le conférencier idéal lorsqu'il s'agit de positionner PostgreSQL dans le contexte actuel.

Conférence

La conférence pgDay Paris aura lieu le 21 avril prochain au siège de La Poste (les détails et la carte sont sur http://pgday.paris/contact/), et nous vous attendons nombreux : Inscrivez-vous dès aujourd'hui !

par dim@tapoueh.org (Dimitri Fontaine) le jeudi 2 avril 2015 à 09h50

mercredi 1 avril 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 22 mars 2015

La conférence PGDay UK aura lieu le 7 juillet 2015 – elle vise les membres de la communauté PostgreSQL anglaise. L'appel à conférenciers expire le 13 avril : http://www.postgresqlusergroup.org.uk

L'appel à conférenciers pour le PostgresOpen 2015, programmé à Dallas (Texas) du 16 au 18 septembre, a été lancé : http://2015.postgresopen.org/callforpapers/

[ndt: 4e rendez-vous du PLUG (Lyon) le 15 avril, avec une présentation de PoWA et des techniques de détection d'index manquants : http://www.meetup.com/PostgreSQL-User-Group-Lyon/events/221188759/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en mars

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Replace insertion sort in contrib/intarray with qsort(). It's all very well to claim that a simplistic sort is fast in easy cases, but O(N^2) in the worst case is not good ... especially if the worst case is as easy to hit as "descending order input". Replace that bit with our standard qsort. Per bug #12866 from Maksym Boguk. Back-patch to all active branches. http://git.postgresql.org/pg/commitdiff/8d1f239003d0245dda636dfa6cf0add13bee69d6
  • Allow foreign tables to participate in inheritance. Foreign tables can now be inheritance children, or parents. Much of the system was already ready for this, but we had to fix a few things of course, mostly in the area of planner and executor handling of row locks. As side effects of this, allow foreign tables to have NOT VALID CHECK constraints (and hence to accept ALTER ... VALIDATE CONSTRAINT), and to accept ALTER SET STORAGE and ALTER SET WITH/WITHOUT OIDS. Continuing to disallow these things would've required bizarre and inconsistent special cases in inheritance behavior. Since foreign tables don't enforce CHECK constraints anyway, a NOT VALID one is a complete no-op, but that doesn't mean we shouldn't allow it. And it's possible that some FDWs might have use for SET STORAGE or SET WITH OIDS, though doubtless they will be no-ops for most. An additional change in support of this is that when a ModifyTable node has multiple target tables, they will all now be explicitly identified in EXPLAIN output, for example: Update on pt1 (cost=0.00..321.05 rows=3541 width=46) Update on pt1 Foreign Update on ft1 Foreign Update on ft2 Update on child3 -> Seq Scan on pt1 (cost=0.00..0.00 rows=1 width=46) -> Foreign Scan on ft1 (cost=100.00..148.03 rows=1170 width=46) -> Foreign Scan on ft2 (cost=100.00..148.03 rows=1170 width=46) -> Seq Scan on child3 (cost=0.00..25.00 rows=1200 width=46) This was done mainly to provide an unambiguous place to attach "Remote SQL" fields, but it is useful for inherited updates even when no foreign tables are involved. Shigeru Hanada and Etsuro Fujita, reviewed by Ashutosh Bapat and Kyotaro Horiguchi, some additional hacking by me http://git.postgresql.org/pg/commitdiff/cb1ca4d800621dcae67ca6c799006de99fa4f0a5

Álvaro Herrera a poussé :

  • Fix out-of-array-bounds compiler warning. Since the array length check is using a post-increment operator, the compiler complains that there's a potential write to one element beyond the end of the array. This is not possible currently: the only path to this function is through pg_get_object_address(), which already verifies that the input array is no more than two elements in length. Still, a bug is a bug. No idea why my compiler doesn't complain about this ... Pointed out by Dead Rasheed and Peter Eisentraut http://git.postgresql.org/pg/commitdiff/a190738457353ddb60743e45972f6fe50a75ee77
  • Support opfamily members in get_object_address. In the spirit of 890192e99af and 4464303405f: have get_object_address understand individual pg_amop and pg_amproc objects. There is no way to refer to such objects directly in the grammar -- rather, they are almost always considered an integral part of the opfamily that contains them. (The only case that deals with them individually is ALTER OPERATOR FAMILY ADD/DROP, which carries the opfamily address separately and thus does not need it to be part of each added/dropped element's address.) In event triggers it becomes possible to become involved with individual amop/amproc elements, and this commit enables pg_get_object_address to do so as well. To make the overall coding simpler, this commit also slightly changes the get_object_address representation for opclasses and opfamilies: instead of having the AM name in the objargs array, I moved it as the first element of the objnames array. This enables the new code to use objargs for the type names used by pg_amop and pg_amproc. Reviewed by: Stephen Frost http://git.postgresql.org/pg/commitdiff/a61fd5334eb1040d0dcec0368702398a5b49152c
  • Rationalize vacuuming options and parameters. We were involving the parser too much in setting up initial vacuuming parameters. This patch moves that responsibility elsewhere to simplify code, and also to make future additions easier. To do this, create a new struct VacuumParams which is filled just prior to vacuum execution, instead of at parse time; for user-invoked vacuuming this is set up in a new function ExecVacuum, while autovacuum sets it up by itself. While at it, add a new member VACOPT_SKIPTOAST to enum VacuumOption, only set by autovacuum, which is used to disable vacuuming of the toast table instead of the old do_toast parameter; this relieves the argument list of vacuum() and some callees a bit. This partially makes up for having added more arguments in an effort to avoid having autovacuum from constructing a VacuumStmt parse node. Author: Michael Paquier. Some tweaks by Álvaro Reviewed by: Robert Haas, Stephen Frost, Álvaro Herrera http://git.postgresql.org/pg/commitdiff/0d831389749a3baaced7b984205b9894a82444b9
  • Setup cursor position for schema-qualified elements. This makes any errors thrown while looking up such schemas report the position of the error. Author: Ryan Kelly Reviewed by: Jeevan Chalke, Tom Lane http://git.postgresql.org/pg/commitdiff/b8d226b4f9691c7afb986dbaaf3f6ff7b203d1b5
  • Install shared libraries to bin/ in Windows under MSVC. Since commit cb4a3b04 we were already doing this for the Cygwin/mingw toolchains, but MSVC had not been updated to do it. At Install.pm time, the Makefile (or GNUmakefile) is inspected, and if a line matching SO_MAJOR_VERSION is found (indicating a shared library is being built), then files with the .dll extension are set to be installed in bin/ rather than lib/, while files with .lib extension are installed in lib/. This makes the MSVC toolchain up to date with cygwin/mingw. This removes ad-hoc hacks that were copying files into bin/ or lib/ manually (libpq.dll in particular was already being copied into bin). So while this is a rather ugly kludge, it's still cleaner than what was there before. Author: Michael Paquier Reviewed by: Asif Naeem http://git.postgresql.org/pg/commitdiff/f9dead5624c63b009fc04229c1e5f660436b747b
  • array_offset() and array_offsets(). These functions return the offset position or positions of a value in an array. Author: Pavel Stěhule Reviewed by: Jim Nasby http://git.postgresql.org/pg/commitdiff/13dbc7a824b3f905904cab51840d37f31a07a9ef

Andres Freund a poussé :

  • Remove docs missed in 51c11a7025. Somehow I misresolved a merge conflict when forward porting Petr's patch leading to a section of the docs remaining... Thankfully Fujii spotted my mistake. http://git.postgresql.org/pg/commitdiff/4559167c6b75be334fabad70d7cc03a38a08d494
  • Use 128-bit math to accelerate some aggregation functions. On platforms where we support 128bit integers, use them to implement faster transition functions for sum(int8), avg(int8), var_*(int2/int4),stdev_*(int2/int4). Where not supported continue to use numeric as a transition type. In some synthetic benchmarks this has been shown to provide significant speedups. Bumps catversion. Discussion: 544BB5F1.50709@proxel.se Author: Andreas Karlsson Reviewed-By: Peter Geoghegan, Petr Jelinek, Andres Freund, Oskari Saarenmaa, David Rowley http://git.postgresql.org/pg/commitdiff/959277a4f579da5243968c750069570a58e92b38
  • Add, optional, support for 128bit integers. We will, for the foreseeable future, not expose 128 bit datatypes to SQL. But being able to use 128bit math will allow us, in a later patch, to use 128bit accumulators for some aggregates; leading to noticeable speedups over using numeric. So far we only detect a gcc/clang extension that supports 128bit math, but no 128bit literals, and no *printf support. We might want to expand this in the future to further compilers; if there are any that that provide similar support. Discussion: 544BB5F1.50709@proxel.se Author: Andreas Karlsson, with significant editorializing by me Reviewed-By: Peter Geoghegan, Oskari Saarenmaa http://git.postgresql.org/pg/commitdiff/8122e1437e332e156d971a0274879b0ee76e488a
  • Fix minor copy & pasto in the int128 accumulator patch. It's unlikely that using PG_GETARG_INT16 instead of PG_GETARG_INT32 in this pace can cause actual problems, but this still should be fixed. http://git.postgresql.org/pg/commitdiff/59b0a98af07cf8decfe94739f92bf18ebb34ffc6

Bruce Momjian a poussé :

Robert Haas a poussé :

  • Fix status reporting for terminated bgworkers that were never started. Previously, GetBackgroundWorkerPid() would return BGWH_NOT_YET_STARTED if the slot used for the worker registration had not been reused by unrelated activity, and BGWH_STOPPED if it had. Either way, a process that had requested notification when the state of one of its background workers changed did not receive such notifications. Fix things so that GetBackgroundWorkerPid() always returns BGWH_STOPPED in this situation, so that we do not erroneously give waiters the impression that the worker will eventually be started; and send notifications just as we would if the process terminated after having been started, so that it's possible to wait for the postmaster to process a worker termination request without polling. Discovered by Amit Kapila during testing of parallel sequential scan. Analysis and fix by me. Back-patch to 9.4; there may not be anyone relying on this interface yet, but if anyone is, the new behavior is a clear improvement. http://git.postgresql.org/pg/commitdiff/bf740ce9e5d82612889d131f34c079215973ca00
  • Add flags argument to dsm_create. Right now, there's only one flag, DSM_CREATE_NULL_IF_MAXSEGMENTS, which suppresses the error that would normally be thrown when the maximum number of segments already exists, instead returning NULL. It might be useful to add more flags in the future, such as one to ignore allocation errors, but I haven't done that here. http://git.postgresql.org/pg/commitdiff/12968cf4085409c50f70c6643d92befdb34008f6

Stephen Frost a poussé :

  • GetUserId() changes to has_privs_of_role(). The pg_stat and pg_signal-related functions have been using GetUserId() instead of has_privs_of_role() for checking if the current user should be able to see details in pg_stat_activity or signal other processes, requiring a user to do 'SET ROLE' for inheirited roles for a permissions check, unlike other permissions checks. This patch changes that behavior to, instead, act like most other permission checks and use has_privs_of_role(), removing the 'SET ROLE' need. Documentation and error messages updated accordingly. Per discussion with Alvaro, Peter, Adam (though not using Adam's patch), and Robert. Reviewed by Jeevan Chalke. http://git.postgresql.org/pg/commitdiff/bf038899965263dbc4aef2b43c8fdfe6f49b788f

Peter Eisentraut a poussé :

Heikki Linnakangas a poussé :

  • Make pg_xlogdump MSVC build work more like others. Instead of copying xlogreader.c and *desc.c files into the source directory, build them where they are. That's what we do for other binaries that need to compile and link in files from elsewhere in the source tree. The commit history suggests that it was done this way because of issues with older versions of MSVC. I think this should work, but we'll see if the buildfarm complains. http://git.postgresql.org/pg/commitdiff/1933a5bbc883fd2697c42d82ae12f2d585559b23

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Kyotaro HORIGUCHI sent in another revision of a patch to reduce pinning in btree indexes.
  • Kaigai Kouhei sent in two more revisions of a patch to add custom/foreign join APIs.
  • David Rowley sent in another revision of a patch to allow removing inner JOINs under certain conditions.
  • Dean Rasheed sent in another revision of a patch to improve RLS qual pushdown.
  • Álvaro Herrera sent in another revision of a flock of patches to add deparsing utility commands.
  • Adam Brightwell sent in a patch to fix a regression test for sepgsql.
  • Michael Paquier sent in another revision of a patch to add in-core regression tests for recovery.
  • Álvaro Herrera sent in another revision of a patch to add a missing type_schema hint.
  • Álvaro Herrera sent in another revision of a patch to move freeze parameters of VacuumStmt into a separate code path.
  • Robert Haas sent in three more revisions of a patch to add parallel mode and parallel contexts.
  • Julien Tachoires and Andreas Karlsson traded patches to allow toast tables to be moved to a different tablespace.
  • Bruce Momjian and Tom Lane traded patches to prevent post-commit interrupts.
  • Robert Haas sent in another revision of a patch to allow testing parallel safety.
  • Amit Kapila sent in another revision of a patch to implement parallel sequential scan.
  • Michael Paquier sent in another revision of a patch to implement table-level log_autovacuum_min_duration.
  • Michael Paquier sent in a patch to fix an example of a variable referencing itself in example of pgbench.sgml.
  • David Christensen sent in two revisions of a patch to add a two-arg current_setting() with fallback.
  • Max Filippov sent in a patch to compare the linker and compiler outputs in the case of pthread with their default output.
  • Bruce Momjian sent in a patch to document the fact that repeatable read and serializable transactions see data committed after transaction start.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add shared infrastructure and functional dependencies for the upcoming multivariate statistics feature.
  • Bruce Momjian sent in another revision of a patch to change pg_ctl's default shutdown mode from "smart" to "fast."
  • Bruce Momjian sent in four revisions of a patch to simplify examples of dynamic SQL.
  • Bruce Momjian sent in a patch to remove an extra VACUUM in initdb.
  • Bruce Momjian sent in two more revisions of a patch to enable asciidoc as a psql output format.
  • David Wheeler sent in another revision of a patch to add launchd support.
  • Peter Geoghegan sent in two more revisions of a patch to add sort support for numerics.
  • Bruce Momjian sent in a patch to document more clearly the limitations of NUMERIC.
  • David Rowley sent in another revision of a patch to improve performance for joins where outer side is unique.
  • Peter Eisentraut sent in another revision of a patch to add TRANSFORMs.
  • Tomas Vondra sent in a patch to add two new log_line_prefix escape sequences - %T and %M, doing the same thing as %t and %m, but formatting the value as a number.
  • Andrew (RhodiumToad) Gierth sent in two more revisions of a patch to standardize INT64_MIN INT64_MAX UINT64_MAX.
  • Fabien COELHO sent in a patch to fix pgbench --progress report under (very) low rate.
  • Amit Khandekar sent in a patch to reset background worker crash state.
  • Dmitry Voronin sent in a patch to add some new functions to th SSL contrib module.
  • Pavel Stehule sent in another revision of a patch to add an ASSERT statement to SQL.

par N Bougain le mercredi 1 avril 2015 à 00h17

dimanche 29 mars 2015

Guillaume Lelarge

Version 0.2 de mon livre sur PostgreSQL

La version 0.2 de mon livre est sortie hier. En dehors des ajouts/correctifs dans les précédents chapitres, elle ajoute deux nouveaux chapitres.

Le premier concerne le protocole de communication client/serveur de PostgreSQL. Il permet de bien prendre conscience du dialogue et des possibilités d'échange entre ces deux entités. Le second aborde la question des connexions : comment s'établie une connexion, quels paramètres de configuration existent pour les connexions, comment gérer les connexions, etc.

De plus, le site d-booker, éditeur du livre, publie une interview de l'auteur (donc moi).

Si vous avez lu le livre « PostgreSQL, architectures et notions avancées », j'aimerais beaucoup savoir ce que vous en avez pensé. N'hésitez pas à intervenir sur le forum pour me remonter vos impressions ou tout problème que vous aurez constaté.

par Guillaume Lelarge le dimanche 29 mars 2015 à 16h46

mercredi 25 mars 2015

Rodolphe Quiédeville

PgDay Paris

J'ai la plaisir cette année de participer au comité de sélection du PgDay Paris qui se déroulera le 21 Avril 2015.

Le pgDay Paris est une journée de conférences et d'échanges organisée par la communauté française de PostgreSQL. Un ensemble de présentations en anglais et en français sera proposé, couvrant des sujets techniques ainsi que des retours d'expérience d'utilisation de PostgreSQL en production.

Que vous soyez développeur, administrateur système ou de bases de données ou bien « décideur » (DSI, directeur technique, etc), nous aurons du contenu pour vous !

Les inscriptions à l'événement sont dès maintenant disponibles pour le prix de 65 € pour la journée, qui inclut les pauses cafés et le déjeuner complet sur place.

Les places étant limitées, je vous invite à vous inscrire au plus tôt afin d'être sûr de pouvoir venir.

  • https://www.postgresql.eu/events/register/pgdayparis2015/

Le prix des places est maintenu volontairement bas afin de permettre au plus grand nombre de participer. Cela est rendu possible grâce au soutien des sponsors, et il reste là aussi des places. Alors si vous souhaitez apporter votre contribution au développement de PostgreSQL n'hésitez pas à prendre contact, toutes les coordonnées sont sur le site de l'évènement.

Rendez-vous le 21 !

par Rodolphe Quiédeville le mercredi 25 mars 2015 à 20h18

dimanche 22 mars 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 15 mars 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en Mars

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Fujii Masao a poussé :

  • Add missing "goto err" statements in xlogreader.c. Spotted by Andres Freund. http://git.postgresql.org/pg/commitdiff/c74c04b8aa03f05983f940ee94c86a5cc1945393
  • Fix typo in comment. http://git.postgresql.org/pg/commitdiff/828599acecdb2929f9b66d6f590c2abbc751b58b
  • Suppress maybe-uninitialized compiler warnings. Previously some compilers were thinking that the variables that 57aa5b2 added maybe-uninitialized. Spotted by Andres Freund http://git.postgresql.org/pg/commitdiff/cd6c45cbeec5e21b470e9a8d19e02f60f1a52807
  • Add GUC to enable compression of full page images stored in WAL. When newly-added GUC parameter, wal_compression, is on, the PostgreSQL server compresses a full page image written to WAL when full_page_writes is on or during a base backup. A compressed page image will be decompressed during WAL replay. Turning this parameter on can reduce the WAL volume without increasing the risk of unrecoverable data corruption, but at the cost of some extra CPU spent on the compression during WAL logging and on the decompression during WAL replay. This commit changes the WAL format (so bumping WAL version number) so that the one-byte flag indicating whether a full page image is compressed or not is included in its header information. This means that the commit increases the WAL volume one-byte per a full page image even if WAL compression is not used at all. We can save that one-byte by borrowing one-bit from the existing field like hole_offset in the header and using it as the flag, for example. But which would reduce the code readability and the extensibility of the feature. Per discussion, it's not worth paying those prices to save only one-byte, so we decided to add the one-byte flag to the header. This commit doesn't introduce any new compression algorithm like lz4. Currently a full page image is compressed using the existing PGLZ algorithm. Per discussion, we decided to use it at least in the first version of the feature because there were no performance reports showing that its compression ratio is unacceptably lower than that of other algorithm. Of course, in the future, it's worth considering the support of other compression algorithm for the better compression. Rahila Syed and Michael Paquier, reviewed in various versions by myself, Andres Freund, Robert Haas, Abhijit Menon-Sen and many others. http://git.postgresql.org/pg/commitdiff/57aa5b2bb11a4dbfdfc0f92370e0742ae5aa367b

Michael Meskes a poussé :

Heikki Linnakangas a poussé :

Robert Haas a poussé :

  • Fix handling of sortKeys field in Tuplesortstate. Commit 5cefbf5a6c4466ac6b1cc2a4316b4eba9108c802 introduced an assumption that this field would always be non-NULL when doing a merge pass, but that's not true. Without this fix, you can crash the server by building a hash index that is sufficiently large relative to maintenance_work_mem, or by triggering a large datum sort. Commit 5ea86e6e65dd2da3e9a3464484985d48328e7fe3 changed the comments for that field to say that it would be set in all cases except for the hash index case, but that wasn't (and still isn't) true. The datum-sort failure was spotted by Tomas Vondra; initial analysis of that failure was by Peter Geoghegan. The remaining issues were spotted by me during review of the surrounding code, and the patch is all my fault. http://git.postgresql.org/pg/commitdiff/2720e96a9bd58b2af177c714c0c5402773e1cf30
  • Allow named parameters to be specified using => in addition to := SQL has standardized on => as the use of to specify named parameters, and we've wanted for many years to support the same syntax ourselves, but this has been complicated by the possible use of => as an operator name. In PostgreSQL 9.0, we began emitting a warning when an operator named => was defined, and in PostgreSQL 9.2, we stopped shipping a =>(text, text) operator as part of hstore. By the time the next major version of PostgreSQL is released, => will have been deprecated for a full five years, so hopefully there won't be too many people still relying on it. We continue to support := for compatibility with previous PostgreSQL releases. Pavel Stehule, reviewed by Petr Jelinek, with a few documentation tweaks by me. http://git.postgresql.org/pg/commitdiff/865f14a2d31af23a05bbf2df04c274629c5d5c4d
  • Suggest to the user the column they may have meant to reference. Error messages informing the user that no such column exists can sometimes provoke a perplexed response. This often happens due to a subtle typo in the column name or, perhaps less likely, in the alias name. To speed discovery of what the real issue is in such cases, we'll now search the range table for approximate matches. If there are one or two such matches that are good enough to think that they might be what the user intended to type, and better than all other approximate matches, we'll issue a hint suggesting that the user might have intended to reference those columns. Peter Geoghegan and Robert Haas http://git.postgresql.org/pg/commitdiff/e529cd4ffa605c6f14f1391af5559b3a44da0336
  • sepgsql: Improve error message when unsupported object type is labeled. KaiGai Kohei, reviewed by Álvaro Herrera and myself http://git.postgresql.org/pg/commitdiff/e96b7c6b9fc4d148a22588894245416b63743368
  • Require non-NULL pstate for all addRangeTableEntryFor* functions. Per discussion, it's better to have a consistent coding rule here. Michael Paquier, per a node from Greg Stark referencing an old post from Tom Lane. http://git.postgresql.org/pg/commitdiff/bc93ac12c2544b6b3a68b6cb0282e0828fa14a34
  • Document the new custom scan APIs. These APIs changed somewhat subsequent to the initial commit, and may change further in the future, but let's document what we have today. KaiGai Kohei and Robert Haas, reviewed by Tom Lane and Thom Brown http://git.postgresql.org/pg/commitdiff/82fe8b1119e4187f3d991564274607b0b4089aca

Álvaro Herrera a poussé :

  • Allow CURRENT/SESSION_USER to be used in certain commands. Commands such as ALTER USER, ALTER GROUP, ALTER ROLE, GRANT, and the various ALTER OBJECT / OWNER TO, as well as ad-hoc clauses related to roles such as the AUTHORIZATION clause of CREATE SCHEMA, the FOR clause of CREATE USER MAPPING, and the FOR ROLE clause of ALTER DEFAULT PRIVILEGES can now take the keywords CURRENT_USER and SESSION_USER as user specifiers in place of an explicit user name. This commit also fixes some quite ugly handling of special standards- mandated syntax in CREATE USER MAPPING, which in particular would fail to work in presence of a role named "current_user". The special role specifiers PUBLIC and NONE also have more consistent handling now. Also take the opportunity to add location tracking to user specifiers. Authors: Kyotaro Horiguchi. Heavily reworked by Álvaro Herrera. Reviewed by: Rushabh Lathia, Adam Brightwell, Marti Raudsepp. http://git.postgresql.org/pg/commitdiff/31eae6028eca4365e7165f5f33fee1ed0486aee0
  • Fix crasher bugs in previous commit. ALTER DEFAULT PRIVILEGES was trying to decode the list of roles in the FOR clause as a list of names rather than of RoleSpecs; and the IN clause in CREATE ROLE was doing the same thing. This was evidenced by crashes on some buildfarm machines, though on my platform this doesn't cause a failure by mere chance; I can reproduce the failures only by adding some padding in struct RoleSpecs. Fix by dereferencing those lists as being of RoleSpecs, not string Values. http://git.postgresql.org/pg/commitdiff/e3f1c24b992acb88e4ccf33118640aee4b11dd47
  • Keep CommitTs module in sync in standby and master. We allow this module to be turned off on restarts, so a restart time check is enough to activate or deactivate the module; however, if there is a standby replaying WAL emitted from a master which is restarted, but the standby isn't, the state in the standby becomes inconsistent and can easily be crashed. Fix by activating and deactivating the module during WAL replay on parameter change as well as on system start. Problem reported by Fujii Masao in http://www.postgresql.org/message-id/CAHGQGwFhJ3CnHo1CELEfay18yg_RA-XZT-7D8NuWUoYSZ90r4Q@mail.gmail.com Author: Petr Jelínek http://git.postgresql.org/pg/commitdiff/4f3924d9cd438ba4e6fd639460f8c859c65d45a3
  • Fix stray sentence fragment in shared_preload_libraries documentation. The introduction in the Shared Library Preloading section already instructs the user to separate multiple library names with commas, so just remove the fragment from here. Author: Dagfinn Ilmari Mannsåker http://git.postgresql.org/pg/commitdiff/bb7b35caf78de80d2ff1643d042e62a71f83abbb
  • Move BRIN page type to page's last two bytes. ... which is the usual convention among AMs, so that pg_filedump and similar utilities can tell apart pages of different AMs. It was also the intent of the original code, but I failed to realize that alignment considerations would move the whole thing to the previous-to-last word in the page. The new definition of the associated macro makes surrounding code a bit leaner, too. Per note from Heikki at http://www.postgresql.org/message-id/546A16EF.9070005@vmware.com http://git.postgresql.org/pg/commitdiff/e491bd2ee34860b14ff18abc5602f9aa5b197a2d
  • Refactor Mkvcbuild.pm to facilitate modules migrations. This is in preparation to "upgrade" some modules from contrib/ to src/bin/, per discussion. Author: Michael Paquier http://git.postgresql.org/pg/commitdiff/66ece312f99f384bd33e4342580e78b0eebf0e74
  • Support user mappings in get_object_address. Since commit 72dd233d3ef we were trying to obtain object addressing information in sql_drop event triggers, but that caused failures when the drops involved user mappings. This addition enables that to work again. Naturally, pg_get_object_address can work with these objects now, too. I toyed with the idea of removing DropUserMappingStmt as a node and using DropStmt instead in the DropUserMappingStmt grammar production, but that didn't go very well: for one thing the messages thrown by the specific code are specialized (you get "server not found" if you specify the wrong server, instead of a generic "user mapping for ... not found" which you'd get it we were to merge this with RemoveObjects --- unless we added even more special cases). For another thing, it would require to pass RoleSpec nodes through the objname/objargs representation used by RemoveObjects, which works in isolation, but gets messy when pg_get_object_address is involved. So I dropped this part for now. Reviewed by Stephen Frost. http://git.postgresql.org/pg/commitdiff/890192e99af5db1d15d5bb73f3f1044faa1d2758
  • Fix libpq test expected output file. Evidently, this test is not run very frequently ... http://git.postgresql.org/pg/commitdiff/d4d7777548ed3ea2ca579003e37f9df4d0e0ab9e
  • Support default ACLs in get_object_address. In the spirit of 890192e99af, this time add support for the things living in the pg_default_acl catalog. These are not really "objects", but they show up as such in event triggers. There is no "DROP DEFAULT PRIVILEGES" or similar command, so it doesn't look like the new representation given would be useful anywhere else, so I didn't try to use it outside objectaddress.c. (That might be a bug in itself, but that would be material for another commit.) Reviewed by Stephen Frost. http://git.postgresql.org/pg/commitdiff/4464303405f1f886d63f8316386621cd7436c5d6

Tom Lane a poussé :

  • Clean up the mess from => patch. Commit 865f14a2d31af23a05bbf2df04c274629c5d5c4d was quite a few bricks shy of a load: psql, ecpg, and plpgsql were all left out-of-step with the core lexer. Of these only the last was likely to be a fatal problem; but still, a minimal amount of grepping, or even just reading the comments adjacent to the places that were changed, would have found the other places that needed to be changed. http://git.postgresql.org/pg/commitdiff/2fbb286647fac2014abdf2fbf6c7b4134be91602
  • Allocate ParamListInfo once per plpgsql function, not once per expression. setup_param_list() was allocating a fresh ParamListInfo for each query or expression evaluation requested by a plpgsql function. There was probably once good reason to do it like that, but for a long time we've had a convention that there's a one-to-one mapping between the function's PLpgSQL_datum array and the ParamListInfo slots, which means that a single ParamListInfo can serve all the function's evaluation requests: the data that would need to be passed is the same anyway. In this patch, we retain the pattern of zeroing out the ParamListInfo contents during each setup_param_list() call, because some of the slots may be stale and we don't know exactly which ones. So this patch only saves a palloc/pfree per evaluation cycle and nothing more; still, that seems to be good for a couple percent overall speedup on simple-arithmetic type statements. In future, though, we might be able to improve matters still more by managing the param array contents more carefully. Also, unify the former use of estate->cur_expr with that of paramLI->parserSetupArg; they both were used to point to the active expression, so we can combine the variables into just one. http://git.postgresql.org/pg/commitdiff/21dcda2713656a7483e3280ac9d2ada20a87a9a9
  • Make operator precedence follow the SQL standard more closely. While the SQL standard is pretty vague on the overall topic of operator precedence (because it never presents a unified BNF for all expressions), it does seem reasonable to conclude from the spec for <boolean value expression> that OR has the lowest precedence, then AND, then NOT, then IS tests, then the six standard comparison operators, then everything else (since any non-boolean operator in a WHERE clause would need to be an argument of one of these). We were only sort of on board with that: most notably, while "<" ">" and "=" had properly low precedence, "<=" ">=" and "<>" were treated as generic operators and so had significantly higher precedence. And "IS" tests were even higher precedence than those, which is very clearly wrong per spec. Another problem was that "foo NOT SOMETHING bar" constructs, such as "x NOT LIKE y", were treated inconsistently because of a bison implementation artifact: they had the documented precedence with respect to operators to their right, but behaved like NOT (i.e., very low priority) with respect to operators to their left. Fixing the precedence issues is just a small matter of rearranging the precedence declarations in gram.y, except for the NOT problem, which requires adding an additional lookahead case in base_yylex() so that we can attach a different token precedence to NOT LIKE and allied two-word operators. The bulk of this patch is not the bug fix per se, but adding logic to parse_expr.c to allow giving warnings if an expression has changed meaning because of these precedence changes. These warnings are off by default and are enabled by the new GUC operator_precedence_warning. It's believed that very few applications will be affected by these changes, but it was agreed that a warning mechanism is essential to help debug any that are. http://git.postgresql.org/pg/commitdiff/c6b3c939b7e0f1d35f4ed4996e71420a993810d2
  • Improve planner's cost estimation in the presence of semijoins. If we have a semijoin, say SELECT * FROM x WHERE x1 IN (SELECT y1 FROM y) and we're estimating the cost of a parameterized indexscan on x, the number of repetitions of the indexscan should not be taken as the size of y; it'll really only be the number of distinct values of y1, because the only valid plan with y on the outside of a nestloop would require y to be unique-ified before joining it to x. Most of the time this doesn't make that much difference, but sometimes it can lead to drastically underestimating the cost of the indexscan and hence choosing a bad plan, as pointed out by David Kubečka. Fixing this is a bit difficult because parameterized indexscans are costed out quite early in the planning process, before we have the information that would be needed to call estimate_num_groups() and thereby estimate the number of distinct values of the join column(s). However we can move the code that extracts a semijoin RHS's unique-ification columns, so that it's done in initsplan.c rather than on-the-fly in create_unique_path(). That shouldn't make any difference speed-wise and it's really a bit cleaner too. The other bit of information we need is the size of the semijoin RHS, which is easy if it's a single relation (we make those estimates before considering indexscan costs) but problematic if it's a join relation. The solution adopted here is just to use the product of the sizes of the join component rels. That will generally be an overestimate, but since estimate_num_groups() only uses this input as a clamp, an overestimate shouldn't hurt us too badly. In any case we don't allow this new logic to produce a value larger than we would have chosen before, so that at worst an overestimate leaves us no wiser than we were before. http://git.postgresql.org/pg/commitdiff/b55722692ba0ceb934bb32bcddb562e2120f43dd
  • Fix old bug in get_loop_count(). While poking at David Kubečka's issue I noticed an ancient logic error in get_loop_count(): it used 1.0 as a "no data yet" indicator, but since that is actually a valid rowcount estimate, this doesn't work. If we have one input relation with 1.0 as rowcount and then another one with a larger rowcount, we should use 1.0 as the result, but we picked the larger rowcount instead. (I think when I coded this, I recognized the conflict, but mistakenly thought that the logic would pick the desired count anyway.) Fixing this changed the plan for one existing regression test case. Since the point of that test is to exercise creation of a particular shape of nestloop plan, I tweaked the query a little bit so it still results in the same plan choice. This is definitely a bug, but I'm hesitant to back-patch since it might change plan choices unexpectedly, and anyway failure to implement a heuristic precisely as intended is a pretty low-grade bug. http://git.postgresql.org/pg/commitdiff/b746d0c32d4fe749c8d39ccb09d8f0fb38bcc197
  • Support flattening of empty-FROM subqueries and one-row VALUES tables. We can't handle this in the general case due to limitations of the planner's data representations; but we can allow it in many useful cases, by being careful to flatten only when we are pulling a single-row subquery up into a FROM (or, equivalently, inner JOIN) node that will still have at least one remaining relation child. Per discussion of an example from Kyotaro Horiguchi. http://git.postgresql.org/pg/commitdiff/f4abd0241de20d5d6a79b84992b9e88603d44134
  • Ensure tableoid reads correctly in EvalPlanQual-manufactured tuples. The ROW_MARK_COPY path in EvalPlanQualFetchRowMarks() was just setting tableoid to InvalidOid, I think on the assumption that the referenced RTE must be a subquery or other case without a meaningful OID. However, foreign tables also use this code path, and they do have meaningful table OIDs; so failure to set the tuple field can lead to user-visible misbehavior. Fix that by fetching the appropriate OID from the range table. There's still an issue about whether CTID can ever have a meaningful value in this case; at least with postgres_fdw foreign tables, it does. But that is a different problem that seems to require a significantly different patch --- it's debatable whether postgres_fdw really wants to use this code path at all. Simplified version of a patch by Etsuro Fujita, who also noted the problem to begin with. The issue can be demonstrated in all versions having FDWs, so back-patch to 9.1. http://git.postgresql.org/pg/commitdiff/443fd0540e298b621be6748dead1fb444556e0d0
  • Improve documentation of bt_page_items(). Explain some of the funny conventions used in btree page items. Peter Geoghegan and Jeff Janes http://git.postgresql.org/pg/commitdiff/ebc0f5e01d2f4b7d7c3307fd4d40498f6b120898
  • Remove workaround for ancient incompatibility between readline and libedit. GNU readline defines the return value of write_history() as "zero if OK, else an errno code". libedit's version of that function used to have a different definition (to wit, "-1 if error, else the number of lines written to the file"). We tried to work around that by checking whether errno had become nonzero, but this method has never been kosher according to the published API of either library. It's reportedly completely broken in recent Ubuntu releases: psql bleats about "No such file or directory" when saving ~/.psql_history, even though the write worked fine. However, libedit has been following the readline definition since somewhere around 2006, so it seems all right to finally break compatibility with ancient libedit releases and trust that the return value is what readline specifies. (I'm not sure when the various Linux distributions incorporated this fix, but I did find that OS X has been shipping fixed versions since 10.5/Leopard.) If anyone is still using such an ancient libedit, they will find that psql complains it can't write ~/.psql_history at exit, even when the file was written correctly. This is no worse than the behavior we're fixing for current releases. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/df9ebf1eeaf98481e00cd77bf6056d42310731b7
  • Build src/port/dirmod.c only on Windows. Since commit ba7c5975adea74c6f17bdb0e0427ad85962092a2, port/dirmod.c has contained only Windows-specific functions. Most platforms don't seem to mind uselessly building an empty file, but OS X for one issues warnings. Hence, treat dirmod.c as a Windows-specific file selected by configure rather than one that's always built. We can revert this change if dirmod.c ever gains any non-Windows functionality again. Back-patch to 9.4 where the mentioned commit appeared. http://git.postgresql.org/pg/commitdiff/91f4a5a976500517e492320e389342d7436cf9d4
  • Remove obsolete comment. Obsoleted by commit 21dcda2713656a7483e3280ac9d2ada20a87a9a9, but I missed seeing the cross-reference in the comments for exec_eval_integer(). Also improve the cross-reference in the comments for exec_eval_cleanup(). http://git.postgresql.org/pg/commitdiff/5ff683962ef9a953eeb17ea58d678f0c4ca189ae
  • Add missing documentation for PGC_SU_BACKEND in description of pg_settings. Commit fe550b2ac249af5fbd8e9e19290a4ba43c882f2d missed updating this list of the PGC_XXX values, which in hindsight is not so surprising because catalogs.sgml is not a place you'd think to look for them. In addition to adding the missing doco, insert the PGC_XXX C enum names in SGML comments, so that grepping for the enum names will find this file. That might spare the next person similar embarrassment. Spotted by Magnus Hagander. http://git.postgresql.org/pg/commitdiff/d1e9214e4ff5604357d0155467f105825a9e102c
  • src/port/dirmod.c needs to be built on Cygwin too. Oversight in my commit 91f4a5a976500517e492320e389342d7436cf9d4. Per buildfarm member brolga. http://git.postgresql.org/pg/commitdiff/80089597730f67927293c410914f3e6bf11ca447
  • Move LockClauseStrength, LockWaitPolicy into new file nodes/lockoptions.h. Commit df630b0dd5ea2de52972d456f5978a012436115e moved enum LockWaitPolicy into its very own header file utils/lockwaitpolicy.h, which does not seem like a great idea from here. First, it's still a node-related declaration, and second, a file named like that can never sensibly be used for anything else. I do not think we want to encourage a one-typedef-per-header-file approach. The upcoming foreign table inheritance patch was doubling down on this bad idea by moving enum LockClauseStrength into its *own* can-never-be-used-for-anything-else file. Instead, let's put them both in a file named nodes/lockoptions.h. (They do seem to need a separate header file because we need them in both parsenodes.h and plannodes.h, and we don't want either of those including the other. Past practice might suggest adding them to nodes/nodes.h, but they don't seem sufficiently globally useful to justify that.) Committed separately since there's no functional change here, just some header-file refactoring. http://git.postgresql.org/pg/commitdiff/9fac5fd741ec17ae24dde6b8e82064f13c148ddf
  • Improve representation of PlanRowMark. This patch fixes two inadequacies of the PlanRowMark representation. First, that the original LockingClauseStrength isn't stored (and cannot be inferred for foreign tables, which always get ROW_MARK_COPY). Since some PlanRowMarks are created out of whole cloth and don't actually have an ancestral RowMarkClause, this requires adding a dummy LCS_NONE value to enum LockingClauseStrength, which is fairly annoying but the alternatives seem worse. This fix allows getting rid of the use of get_parse_rowmark() in FDWs (as per the discussion around commits 462bd95705a0c23b and 8ec8760fc87ecde0), and it simplifies some things elsewhere. Second, that the representation assumed that all child tables in an inheritance hierarchy would use the same RowMarkType. That's true today but will soon not be true. We add an "allMarkTypes" field that identifies the union of mark types used in all a parent table's children, and use that where appropriate (currently, only in preprocess_targetlist()). In passing fix a couple of minor infelicities left over from the SKIP LOCKED patch, notably that _outPlanRowMark still thought waitPolicy is a bool. Catversion bump is required because the numeric values of enum LockingClauseStrength can appear in on-disk rules. Extracted from a much larger patch to support foreign table inheritance; it seemed worth breaking this out, since it's a separable concern. Shigeru Hanada and Etsuro Fujita, somewhat modified by me http://git.postgresql.org/pg/commitdiff/7b8b8a43317e9e59eca8b511b714a0ab7da5f1cb

Andres Freund a poussé :

  • Add macros wrapping all usage of gcc's __attribute__. Until now __attribute__() was defined to be empty for all compilers but gcc. That's problematic because it prevents using it in other compilers; which is necessary e.g. for atomics portability. It's also just generally dubious to do so in a header as widely included as c.h. Instead add pg_attribute_format_arg, pg_attribute_printf, pg_attribute_noreturn macros which are implemented in the compilers that understand them. Also add pg_attribute_noreturn and pg_attribute_packed, but don't provide fallbacks, since they can affect functionality. This means that external code that, possibly unwittingly, relied on __attribute__ defined to be empty on !gcc compilers may now run into warnings or errors on those compilers. But there shouldn't be many occurances of that and it's hard to work around... Discussion: 54B58BA3.8040302@ohmu.fi Author: Oskari Saarenmaa, with some minor changes by me. http://git.postgresql.org/pg/commitdiff/bbfd7edae5aa5ad5553d3c7e102f2e450d4380d4
  • Adjust valgrind suppressions wrt 025c02420. http://git.postgresql.org/pg/commitdiff/241f088f3632814fe9dbd5bcbc509ec42a268d01
  • Increase max_wal_size's default from 128MB to 1GB. The introduction of min_wal_size & max_wal_size in 88e982302684 makes it feasible to increase the default upper bound in checkpoint size. Previously raising the default would lead to a increased disk footprint, even if more segments weren't beneficial. The low default of checkpoint size is one of common performance problem users have thus increasing the default makes sense. Setups where the increase in maximum disk usage is a problem will very likely have to run with a modified configuration anyway. Discussion: 54F4EFB8.40202@agliodbs.com, CA+TgmoZEAgX5oMGJOHVj8L7XOkAe05Gnf45rP40m-K3FhZRVKg@mail.gmail.com Author: Josh Berkus, after a discussion involving lots of people. http://git.postgresql.org/pg/commitdiff/a0f5954af19ddcfea946b15744f2006a789dc4bd
  • Remove pause_at_recovery_target recovery.conf setting. The new recovery_target_action (introduced in aedccb1f6/b8e33a85d4) replaces it's functionality. Having both seems likely to cause more confusion than it saves worry due to the incompatibility. Discussion: 5484FC53.2060903@2ndquadrant.com Author: Petr Jelinek http://git.postgresql.org/pg/commitdiff/51c11a7025ecc10c2b922d60a056ad7c6cf147a5
  • Merge the various forms of transaction commit & abort records. Since 465883b0a two versions of commit records have existed. A compact version that was used when no cache invalidations, smgr unlinks and similar were needed, and a full version that could deal with all that. Additionally the full version was embedded into twophase commit records. That resulted in a measurable reduction in the size of the logged WAL in some workloads. But more recently additions like logical decoding, which e.g. needs information about the database something was executed on, made it applicable in fewer situations. The static split generally made it hard to expand the commit record, because concerns over the size made it hard to add anything to the compact version. Additionally it's not particularly pretty to have twophase.c insert RM_XACT records. Rejigger things so that the commit and abort records only have one form each, including the twophase equivalents. The presence of the various optional (in the sense of not being in every record) pieces is indicated by a bits in the 'xinfo' flag. That flag previously was not included in compact commit records. To prevent an increase in size due to its presence, it's only included if necessary; signalled by a bit in the xl_info bits available for xact.c, similar to heapam.c's XLOG_HEAP_OPMASK/XLOG_HEAP_INIT_PAGE. Twophase commit/aborts are now the same as their normal counterparts. The original transaction's xid is included in an optional data field. This means that commit records generally are smaller, except in the case of a transaction with subtransactions, but no other special cases; the increase there is four bytes, which seems acceptable given that the more common case of not having subtransactions shrank. The savings are especially measurable for twophase commits, which previously always used the full version; but will in practice only infrequently have required that. The motivation for this work are not the space savings and and deduplication though; it's that it makes it easier to extend commit records with additional information. That's just a few lines of code now; without impacting the common case where that information is not needed. Discussion: 20150220152150.GD4149@awork2.anarazel.de, 235610.92468.qm%40web29004.mail.ird.yahoo.com Reviewed-By: Heikki Linnakangas, Simon Riggs http://git.postgresql.org/pg/commitdiff/4f1b890b1377744688e43218f975d3c8b2ae39f8

Peter Eisentraut a poussé :

Tatsuo Ishii a poussé :

  • Fix integer overflow in debug message of walreceiver. The message tries to tell the replication apply delay which fails if the first WAL record is not applied yet. Fix is, instead of telling overflowed minus numeric, showing "N/A" which indicates that the delay data is not yet available. Problem reported by me and patch by Fabrízio de Royes Mello. Back patched to 9.4, 9.3 and 9.2 stable branches (9.1 and 9.0 do not have the debug message). http://git.postgresql.org/pg/commitdiff/364c006c1fba7ba7825fb06ef0166e752546f357

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Michael Paquier sent in two more revisions of a patch to speed up make check-world.
  • Dmitry Voronin sent in another revision of a patch to add some new functions to contrib/sslinfo.
  • Michael Paquier sent in two more revisions of a patch to install shared libs in lib/ and bin/ with MSVC (Was: install libpq.dll in bin directory on Windows / Cygwin.
  • Kaigai Kouhei sent in another revision of a patch to add join push-down support for foreign tables.
  • SAWADA Masahiko sent in three more revisions of a patch to add REINDEX xxx VERBOSE.
  • Andrew (RhodiumToad) Gierth sent in two more revisions of a patch to add GROUPING SETS.
  • Amit Kapila and Amit Langote traded patches to add parallel sequential scan.
  • Kevin Grittner sent in three more revisions of a patch to reduce pinning in btree indexes.
  • Kaigai Kouhei sent in another revision of a patch to add a custom plan API.
  • Kaigai Kouhei and Robert Haas traded patches for the custom join API.
  • Heikki Linnakangas sent in three more revisions of a patch to add pg_rewind.
  • Andreas Karlsson sent in three more revisions of a patch to use 128-bit integers for sum, avg and statistics aggregates.
  • Abhijit Menon-Sen sent in another revision of a patch to recursively fsync PGDATA at startup if needed.
  • Kyotaro HORIGUCHI sent in another revision of a patch to make an effort to send feedback regulary on heavy load.
  • Tom Lane sent in a patch to fix some inefficiencies in PL/pgsql.
  • Robert Haas sent in another revision of a patch to do better at HINTing at a missing column.
  • Michael Paquier and Peter Eisentraut traded flocks of patches intended to move binaries from contrib/ to bin/.
  • Jim Nasby and Pavel Stehule traded patches to add an array_offset function.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add regrole.
  • SAWADA Masahiko sent in another revision of a patch to add a pg_file_settings view.
  • Fabien COELHO sent in another revision of a patch to improve pgbench syntax error messages.
  • Gregory Stark sent in another revision of a patch to provide a catalog view to pg_hba.conf file.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add alter user/role CURRENT_USER.
  • Jozef Mlich sent in a patch to use the tcl binary in the PL/tcl scripts rather than invoking them as shell programs.
  • Marko (johto) Tiikkaja sent in a patch to fix a bug in dumping views with circular dependencies.
  • Stas Kelvich sent in another revision of a patch to add cube extension KNN support.
  • Álvaro Herrera sent in a patch to support opfamily members in get_object_address.
  • Robert Haas sent in another revision of a patch to help assessing parallel-safety.
  • Michael Paquier sent in another revision of a patch to fix the slash leanings in the MSVC build code.
  • Michael Paquier sent in a patch to move freeze parameters of VacuumStmt into a separate structure.
  • Tomas Vondra sent in a patch to fix an xloginsert.c hole_length warning on gcc 4.8.3.
  • Peter Geoghegan sent in another revision of a patch to do datumsort revisions for abbreviated keys.
  • Julien Tachoires sent in two more revisions of a patch to allow toast tables to be moved to a different tablespace.
  • David Rowley sent in another revision of a patch to improve performance for for joins where outer side is unique.
  • Pavel Stehule sent in a patch to add a --strict tablename parameter to pg_dump.
  • Andreas Karlsson sent in a patch to fix support for the --no-tablespaces flag for pg_dump.
  • Tom Lane sent in a patch to remove setup_param_list() overhead for the common case of PLPGSQL_DTYPE_VAR variables (ie, any non-composite type).
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to implement TABLESAMPLE.
  • Andres Freund sent in another revision of a patch to merge compact/non compact commits, and make aborts dynamically sized.
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to create a sequence access method.

par N Bougain le dimanche 22 mars 2015 à 22h28

Adrien Nayrat

Restauration PITR – Part 7

Je profite d’un week end de mauvais temps (m’empêchant de retourner en falaise :( ) pour poursuivre mes articles sur Postgres.

Dans le précédent article : Backup, restauration – Part 6 J’ai évoqué la restauration PITR pour Point In Time Recovery. Je vous encourage vivement à mettre en place ce qui va suivre afin de réduire la perte de données en cas de modification accidentelle sur des enregistrements : Un drop table par exemple.

Pour la mettre en place il vous faut :

  • Une instance Postgres maître.
  • Archivage des WAL, externalisé de préférence.

Je ne vais pas revenir sur la mise en place de l’archivage, je l’ai déjà détaillé dans les précédents articles. Pour vous expliquer cette restauration je vais utiliser un scénario “catastrophe”.

Il est 12H30 et on vous informe qu’une table a été supprimée par erreur sur un serveur de production.

La réplication que vous avez mis en place ne vous est d’aucune aide car l’opération a été reportée sur le réplicat.

Remarque : Avec la version 9.4 il est possible d’avoir un réplicat avec du retard. Avec un peu de chance l’opération de suppression de la table n’est peut être pas encore réalisée sur le réplicat.

Par chance, vous effectuez un backup full toutes les nuits à 1H00 et vous avez mis en place l’archivage des journaux. Leur nettoyage se fait après le backup full grâce à pg_archivecleanup.

Pour être tranquille vous effectuez un checkpoint afin que les derniers journaux soient archivés. Ensuite vous stoppez votre instance Postgres et déplacez le répertoire (au cas où).

Vous restaurez les fichiers du backup full. Il ne vous reste plus qu’a configurer correctement la restauration via le fichier recovery.conf.

Celui-ci devra contenir le paramètre restore_command afin que Postgres récupére les journaux de transactions archivés.

Il devra également contenir un paramètre pour la cible de récupération, Si on ne le spécifie pas le moteur va rejouer les journaux de transaction dans leur totalité, jusqu’au “drop table” fatal. Il est possible de demander la restauration jusqu’à un point de restauration, une date ou un identifiant de transaction.

Pour approfondir : http://docs.postgresqlfr.org/current/recovery-target-settings.html

Revenons à notre scénario, imaginons que nous souhaitons revenir à la dernière transaction jouée avant le drop fatal. En grattant un peu, on peut utiliser l’outil  pg_xlogdump pour lire les journaux de transaction.

/usr/lib/postgresql/9.3/bin/pg_xlogdump 000000010000000300000066
rmgr: Heap        len (rec/tot):     21/   197, tx:    1213547, lsn: 3/66000028, prev 3/650000F0, bkp: 1000, desc: insert: rel 1663/16486/16519; tid 0/2
rmgr: Transaction len (rec/tot):     12/    44, tx:    1213547, lsn: 3/660000F0, prev 3/66000028, bkp: 0000, desc: commit: 2015-03-22 12:23:35.834785 CET
rmgr: Standby     len (rec/tot):     16/    48, tx:    1213548, lsn: 3/66000120, prev 3/660000F0, bkp: 0000, desc: AccessExclusive locks: xid 1213548 db 16486 rel 16519
rmgr: Standby     len (rec/tot):     16/    48, tx:    1213548, lsn: 3/66000150, prev 3/66000120, bkp: 0000, desc: AccessExclusive locks: xid 1213548 db 16486 rel 16522
rmgr: Standby     len (rec/tot):     16/    48, tx:    1213548, lsn: 3/66000180, prev 3/66000150, bkp: 0000, desc: AccessExclusive locks: xid 1213548 db 16486 rel 16524
rmgr: Heap        len (rec/tot):     26/  6642, tx:    1213548, lsn: 3/660001B0, prev 3/66000180, bkp: 1000, desc: delete: rel 1663/16486/11774; tid 7/35 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/  3270, tx:    1213548, lsn: 3/66001BA8, prev 3/660001B0, bkp: 1000, desc: delete: rel 1663/16486/11903; tid 46/44 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/  4774, tx:    1213548, lsn: 3/66002888, prev 3/66001BA8, bkp: 1000, desc: delete: rel 1663/16486/11821; tid 2/26 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/  5730, tx:    1213548, lsn: 3/66003B30, prev 3/66002888, bkp: 1000, desc: delete: rel 1663/16486/11786; tid 42/37 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/    58, tx:    1213548, lsn: 3/660051B0, prev 3/66003B30, bkp: 0000, desc: delete: rel 1663/16486/11786; tid 42/38 KEYS_UPDATED
rmgr: Heap        len (rec/tot):     26/  7902, tx:    1213548, lsn: 3/660051F0, prev 3/660051B0, bkp: 1000, desc: delete: rel 1663/16486/11797; tid 0/80 KEYS_UPDATED

C’est un extrait du résultat de la commande, le première ligne indique un insert avec la transaction N° 1213547, la ligne ne dessous indique un commit. En revanche à la troisième ligne on est passé à la transaction 1213548 avec un lock suivi de plusieurs lignes indiquant un delete. Il s’agit du drop fatal.

On va donc revenir à la transaction N°1213547. Voici le contenu de mon fichier recovery.conf :

restore_command = 'rsync "192.168.1.128:/var/lib/postgresql/9.3/wal_archive/%f" "%p"'

recovery_target_xid = '1213547'
recovery_target_inclusive = 'true'

La directive recovery_target_inclusive indique au moteur s’il inclue la transaction correspondant au N°1213547 ou s’il s’arrête juste avant. Ainsi j’aurais pu spécifier un xid de 1213548  et un target_inclusive à “false” pour obtenir le même résultat.

Ensuite il suffit de redémarrer postgres, voici ce qu’on peut observer dans les logs :

2015-03-22 12:41:17 CET LOG:  database system was interrupted; last known up at 2015-03-22 12:22:59 CET
2015-03-22 12:41:17 CET LOG:  creating missing WAL directory "pg_xlog/archive_status"
2015-03-22 12:41:17 CET LOG:  starting point-in-time recovery to XID 1213547
2015-03-22 12:41:17 CET LOG:  connection received: host=[local]
2015-03-22 12:41:17 CET LOG:  incomplete startup packet
2015-03-22 12:41:18 CET LOG:  connection received: host=[local]
2015-03-22 12:41:18 CET FATAL:  the database system is starting up
2015-03-22 12:41:18 CET LOG:  restored log file "000000010000000300000065" from archive
2015-03-22 12:41:18 CET LOG:  redo starts at 3/65000028
2015-03-22 12:41:18 CET LOG:  consistent recovery state reached at 3/650000F0
2015-03-22 12:41:18 CET LOG:  connection received: host=[local]
2015-03-22 12:41:18 CET FATAL:  the database system is starting up
2015-03-22 12:41:19 CET LOG:  connection received: host=[local]
2015-03-22 12:41:19 CET FATAL:  the database system is starting up
2015-03-22 12:41:19 CET LOG:  restored log file "000000010000000300000066" from archive
2015-03-22 12:41:19 CET LOG:  recovery stopping after commit of transaction 1213547, time 2015-03-22 12:23:35.834785+01
2015-03-22 12:41:19 CET LOG:  redo done at 3/660000F0
2015-03-22 12:41:19 CET LOG:  last completed transaction was at log time 2015-03-22 12:23:35.834785+01
rsync: link_stat "/var/lib/postgresql/9.3/wal_archive/00000002.history" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1536) [Receiver=3.0.9]
2015-03-22 12:41:19 CET LOG:  selected new timeline ID: 2
rsync: link_stat "/var/lib/postgresql/9.3/wal_archive/00000001.history" failed: No such file or directory (2)
2015-03-22 12:41:19 CET LOG:  connection received: host=[local]
2015-03-22 12:41:19 CET FATAL:  the database system is starting up
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1536) [Receiver=3.0.9]

On remarque que le moteur créée le répertoire des journaux de transaction, ensuite la ligne suivante est assez explicite : “starting point-in-time recovery to XID 1213547″. Quelques lignes plus bas on remarque qu’il rejoue les journaux de transaction (“restored log file “000000010000000300000065” from archive”).

Il arrête la restauration jusqu’à la transaction spécifiée :

recovery stopping after commit of transaction 1213547, time 2015-03-22 12:23:35.834785+01

On remarque quelques lignes d’erreur correspondant à la sortie de rsync. Visiblement il cherche à copier plusieurs fichiers en “x.history”.  En regardant dans le répertoire d’archivage on constate que le moteur a créé un fichier 00000002.history contenant :

1       3/66000120      after transaction 1213547

Point important également, on constate que la numérotation des journaux de transaction a changée :

-rw------- 1 postgres postgres 16777216 mars  22 12:23 000000010000000300000065
-rw------- 1 postgres postgres      305 mars  22 12:23 000000010000000300000065.00000028.backup
-rw------- 1 postgres postgres 16777216 mars  22 12:41 000000010000000300000066
-rw------- 1 postgres postgres 16777216 mars  22 12:46 000000020000000300000066
-rw------- 1 postgres postgres       39 mars  22 12:41 00000002.history

Le fichier “000000010000000300000065” correspond au wal lors du pg_basebackup, “000000010000000300000066” correspond au checkpoint que j’ai lancé après l’incident. Après la restauration on peut voir le fichier “000000020000000300000066”. Le huitième caractère est passé de 1 à 2.

En retournant dans les logs du moteur on peut lire la ligne suivante un peu plus bas : “selected new timeline ID: 2″En fait, lors de la restauration, Postgres créée une nouvelle timeline. C’est un concept que j’aborderai plus tard, en attendant je vous invite à voir cette présentation sur le fonctionnement des timeline après une restauration PIRT : https://wiki.postgresql.org/images/e/e5/FOSDEM2013-Timelines.pdf

Bien entendu comme tout système de sauvegarde vous devez jouer la procédure afin de valider qu’il y a pas eu d’erreur ou oubli.

 

FacebookTwitterGoogle+ViadeoPrintEmailGoogle GmailLinkedInPartager

par Adrien Nayrat le dimanche 22 mars 2015 à 12h25

lundi 16 mars 2015

Damien Clochard

70 Nuances de Postgres

Le support du standard SQL/MED a été introduit dans PostgreSQL 9.1 (2011). Quatre ans plus tard, nous avons maintenant plus de 70 Foreign Data Wrappers (FDW) disponibles pour lire et écrire sur tout type de moteur de stockage: base Oracle ou MongoDB, fichiers XML, cluster Hadoop, dépot Git, Twitter, etc. etc. etc.

Il y a quelque jours, pendant le FOSDEM j’ai assisté à la conférence de mon collègue Ronan Dunklau intitulée Foreign Data Wrappers in PostgreSQL : Where are we now ? Ronan est l’auteur de l’extension multicorn et pendant sa présentation je me suis rappelé à quel point multicorn est probablement le projet le plus sous-estimé de la communauté PostgreSQL. En guise d’exercice, j’ai commencé à compter tous les FDW que je connaissais… et j’ai rapidement réalisé qu’il y en avait trop pour ma mémoire de poisson rouge !

Du coup, je suis retourné sur la liste de FDW du wiki PostgreSQL. J’ai commencé à mettre à jour et reformatter le catalogue de tous les wrappers existants et j’ai abouti à une liste de plus de 70 wrappers pour PostgreSQL…

Quelques leçons à en tirer

Pendant cet inventaire, j’ai découvert quelques points intéressants:

  • Tous les SGBD majeurs sont couverts à l’exception de DB2. Bizarrement DB2 est aussi le seul SGBD majeur ayant une implémentation du standard SQL/MED. Difficile de dire si ces deux faits sont liés ou pas…

  • Un tiers des FDW sont écrits en Python et basés sur Multicorn. Les autres sont des wrappers “natifs” écrits en C. Assez logiquement les wrappers en C sont prvilégiés pour les connecteurs de SGBD (ODBC, Oracle, etc.) pour des raisons de performance. En parallèle Multicorn est utilisé majoritairement pour les interroger des web services (S3 storage,RSS files, etc.) et des formats de données specifiques (fichiers de genotype, GeoJSON, etc.)

  • Je ne pense pas que c’était le but initial mais les Foreign Data Wrappers sont aussi un vecteur d’innovation. Certains connecteurs sont détournés de leur usage premier pour développer de nouvelles fonctionnalités. Par exemple, CitusDB a publié une base de données orientée colonnes basée sur un FDW, d’autres wrappers vont dans des directions encore plus exotiques comme la collecte de stats au niveau OS, de l’accéleration GPU pour les seq scans ou un moteur traitement parallèle de requêtes… La plupart de ces projets ne conviennent pas pour de la production bien sur. Cependant cela démontre qu’il est aisé d’implémenter un PoC et à quel point cette technologie est versatile.

  • Le réseau de distribution PGXN semble plutot méconnu. Seulement 20% des wrappers sont disponibles via PGXN. Ou peut-être que les développeurs sont trop paresseux pour créer un paquet pour leur extension ?

  • Le succès de l’implémentation d’ SQL/MED dans PostgreSQL a toutefois des inconvénients. A vue de nez, il y aura plus de 100 wrappers disponibles à la fin 2015. Pour certaines sources de données comme mongodb ou LDAP, il y a déjà plusieurs wrappers différents, sans parler de la cohorte de connecteurs Hadoop :) Sur le long terme, les utilisateurs de PostgreSQL auront du mal à déterminer quels sont les wrappers maintenus et lesquels sont abandonnés… La page de wiki est une tentative pour répondre à cette question mais il y a surement d’autres solutions pour fournir des informations précises et exhaustives aux utilisateurs. Peut-être qu’il faudrait un data Wrapper qui pourrait lister tous les data wrappers :)

The next big thing : IMPORT FOREIGN SCHEMA

De plus, il reste une limitation majeure à l’implémentation actuelle : les meta données. Actuellement, il n’y a pas de moyen simple d’utiliser les (éventuelles) capacités d’introspection de la source de données distante. En clair : vous devez créer toutes vos tables externes une par une. Lorsque l’on veut se connecter à toutes les tables d’une base distantes, c’est répétitif et il y a un risque d’erreurs. Si vous vous connectez à une base PostgreSQL distante, vous pouvez utiliser des astuces mais clairement c’est une tache qui devrait être faite au niveau du connecteur lui-même.

Heureusement voici la commande IMPORT FOREIGN SCHEMA ! Cette nouvelle fonctionnalité a été écrite par Ronan Dunklau, Michael Paquier et Tom Lane. Vous pouvez lire une démo rapide sur le blog de Michael. Cette commande sera disponible dans la version 9.5.

C’est une amélioration énorme ! Avec cet IMPORT et les douzaines de wrappers disponibles, PostgreSQL est en passe de devenir une plateforme d’intégration de données unique et le besoin d’utiliser des outils d’ETL externes devient de plus en plus faible.

PostgreSQL as a Data Integration Plateform

Liens :

EDIT : Merci à Eric Pommereau et Guénolé Marquier pour la relecture !

lundi 16 mars 2015 à 11h17

jeudi 12 mars 2015

Nicolas Thauvin

pg_back le script de base pour sauvegarder PostgreSQL

Il y a fort longtemps, et c'est ma première contribution relative à PostgreSQL, j'ai écrit un script de backup qui dump tout un serveur PostgreSQL avec pg_dump et pg_dumpall. Il s'agit de pg_back.

Cela peut paraître curieux de publier un simple script de sauvegarde que tout DBA PostgreSQL a écrit dans sa vie et sait écrire par cœur. Surtout qu'on le réécrit en permanence ce script, pour ajuster des chemins, des cas particuliers du serveur à sauvegarder et de l'environnement où l'on sauvegarde...

En bien justement, c'est parce qu'on le réécrit tout le temps que pg_back fait gagner du temps. Il est simple et court, facilement lisible, c'est du shell : tout ce qu'il faut pour en faire une bonne base pour créer un script de sauvegarde adapté. Quand on l'utilise comme patron pour en faire un outil plus évolué, on gagne du temps.

Justement rajouter du code pour l'adapter peut se faire au début. Si on n'a pas envie d'utiliser le fichier de configuration, on adapte la liste de variables au début du script, quitte à en rajouter.

L'autre endroit intéressant c'est tout à la fin, avant le exit, on peut rajouter tout ce qu'il faut pour externaliser ses sauvegardes.

C'est par ici.

jeudi 12 mars 2015 à 15h24

mardi 10 mars 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 8 mars 2015

L'assemblée constituante du futur PostgreSQL Users Group suisse (SwissPUG) aura lieu vendredi 10 avril 2015 : http://www.swisspug.org

Il y aura une session PostgreSQL lors de la conférence sur les BDD (DTCC) le 18 avril 2015 à Beijing : http://dtcc.it168.com/list_jiabin.html

Le pgDay Paris aura lieu le 21 avril 2015 : http://pgday.paris/

Les nouveautés des produits dérivés

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Stephen Frost a poussé :

  • Fix pg_dump handling of extension config tables Since 9.1, we've provided extensions with a way to denote "configuration" tables- tables created by an extension which the user may modify. By marking these as "configuration" tables, the extension is asking for the data in these tables to be pg_dump'd (tables which are not marked in this way are assumed to be entirely handled during CREATE EXTENSION and are not included at all in a pg_dump). Unfortunately, pg_dump neglected to consider foreign key relationships between extension configuration tables and therefore could end up trying to reload the data in an order which would cause FK violations. This patch teaches pg_dump about these dependencies, so that the data dumped out is done so in the best order possible. Note that there's no way to handle circular dependencies, but those have yet to be seen in the wild. The release notes for this should include a caution to users that existing pg_dump-based backups may be invalid due to this issue. The data is all there, but restoring from it will require extracting the data for the configuration tables and then loading them in the correct order by hand. Discussed initially back in bug #6738, more recently brought up by Gilles Darold, who provided an initial patch which was further reworked by Michael Paquier. Further modifications and documentation updates by me. Back-patch to 9.1 where we added the concept of extension configuration tables. http://git.postgresql.org/pg/commitdiff/ebd092bc2a07787b31b249d62033b9c8140a5d85

Robert Haas a poussé :

  • pgbench: Add a real expression syntax to \set. Previously, you could do \set variable operand1 operator operand2, but nothing more complicated. Now, you can \set variable expression, which makes it much simpler to do multi-step calculations here. This also adds support for the modulo operator (%), with the same semantics as in C. Robert Haas and Fabien Coelho, reviewed by Álvaro Herrera and Stephen Frost http://git.postgresql.org/pg/commitdiff/878fdcb843e087cc1cdeadc987d6ef55202ddd04
  • pgbench: Fix mistakes in Makefile. My commit 878fdcb843e087cc1cdeadc987d6ef55202ddd04 was not quite right. Tom Lane pointed out one of the mistakes fixed here, and I noticed the other myself while reviewing what I'd committed. http://git.postgresql.org/pg/commitdiff/e5f36902495d0c8d5dee9a5f43fb45d44540f795
  • Remove residual NULL-pstate handling in addRangeTableEntry. Passing a NULL pstate wouldn't actually work, because isLockedRefname() isn't prepared to cope with it; and there hasn't been any in-core code that tries in over a decade. So just remove the residual NULL handling. Spotted by Coverity; analysis and patch by Michael Paquier. http://git.postgresql.org/pg/commitdiff/5223ddacdc737b401ed58184e321f354bdf46686

Tom Lane a poussé :

  • Fix busted markup. Evidently from commit 878fdcb843e087cc1cdeadc987d6ef55202ddd04. Per buildfarm. http://git.postgresql.org/pg/commitdiff/d1479011744d80d80c669b5bd64dc32187f26c1e
  • Reduce json <=> jsonb casts from explicit-only to assignment level. There's no reason to make users write an explicit cast to store a json value in a jsonb column or vice versa. We could probably even make these implicit, but that might open us up to problems with ambiguous function calls, so for now just do this. http://git.postgresql.org/pg/commitdiff/b67f1ce181910e012b3a8ec7a35ba20a48247757
  • Fix long-obsolete code for separating filter conditions in cost_index(). This code relied on pointer equality to identify which restriction clauses also appear in the indexquals (and, therefore, don't need to be applied as simple filter conditions). That was okay once upon a time, years ago, before we introduced the equivalence-class machinery. Now there's about a 50-50 chance that an equality clause appearing in the indexquals will be the mirror image (commutator) of its mate in the restriction list. When that happens, we'd erroneously think that the clause would be re-evaluated at each visited row, and therefore inflate the cost estimate for the indexscan by the clause's cost. Add some logic to catch this case. It seems to me that it continues not to be worthwhile to expend the extra predicate-proof work that createplan.c will do on the finally-selected plan, but this case is common enough and cheap enough to handle that we should do so. This will make a small difference (about one cpu_operator_cost per row) in simple cases; but in situations where there's an expensive function in the indexquals, it can make a very large difference, as seen in recent example from Jeff Janes. This is a long-standing bug, but I'm hesitant to back-patch because of the possibility of destabilizing plan choices that people may be happy with. http://git.postgresql.org/pg/commitdiff/497bac7d290df13d8b00ba48653a96015ff4741b
  • Fix cost estimation for indexscans on expensive indexed expressions. genericcostestimate() and friends used the cost of the entire indexqual expressions as the charge for initial evaluation of indexscan arguments. But of course the index column is not evaluated, only the other side of the qual expression, so this was a bad overestimate if the index column was an expensive expression. To fix, refactor the logic in this area so that there's a single routine charged with deconstructing index quals and figuring out what is the index column and what is the comparison expression. This is more or less free in the case of btree indexes, since btcostestimate() was doing equivalent deconstruction already. It probably adds a bit of new overhead in the cases of other index types, but not a lot. (In the case of GIN I think I saved something by getting rid of code that wasn't aware that the index column associations were already available "for free".) Per recent gripe from Jeff Janes. Arguably this is a bug fix, but I'm hesitant to back-patch because of the possibility of destabilizing plan choices that people may be happy with. http://git.postgresql.org/pg/commitdiff/b9896198cfbc1b0cd0c631d2af72ffe34bd4c7e5
  • Use standard casting mechanism to convert types in plpgsql, when possible. plpgsql's historical method for converting datatypes during assignments was to apply the source type's output function and then the destination type's input function. Aside from being miserably inefficient in most cases, this method failed outright in many cases where a user might expect it to work; an example is that "declare x int; ... x := 3.9;" would fail, not round the value to 4. Instead, let's convert by applying the appropriate assignment cast whenever there is one. To avoid breaking compatibility unnecessarily, fall back to the I/O conversion method if there is no assignment cast. So far as I can tell, there is just one case where this method produces a different result than the old code in a case where the old code would not have thrown an error. That is assignment of a boolean value to a string variable (type text, varchar, or bpchar); the old way gave boolean's output representation, ie 't'/'f', while the new way follows the behavior of the bool-to-text cast and so gives 'true' or 'false'. This will need to be called out as an incompatibility in the 9.5 release notes. Aside from handling many conversion cases more sanely, this method is often significantly faster than the old way. In part that's because of more effective caching of the conversion info. http://git.postgresql.org/pg/commitdiff/1345cc67bbb014209714af32b5681b1e11eaf964
  • Need to special-case RECORD as well as UNKNOWN in plpgsql's casting logic. This is because can_coerce_type thinks that RECORD can be cast to any composite type, but coerce_record_to_complex only works for inputs that are RowExprs or whole-row Vars, so we get a hard failure on a CaseTestExpr. Perhaps these corner cases ought to be fixed so that coerce_to_target_type actually returns NULL as per its specification, rather than failing ... but for the moment an extra check here is the path of least resistance. http://git.postgresql.org/pg/commitdiff/45f2c2fc4e4adcf75cd689e18dab77ebe622fc2e
  • Change plpgsql's cast cache to consider source typmod as significant. I had thought that there was no need to maintain separate cache entries for different source typmods, but further experimentation shows that there is an advantage to doing so in some cases. In particular, if a domain has a typmod (say, "CREATE DOMAIN d AS numeric(20,0)"), failing to notice the source typmod leads to applying a length-coercion step even when the source has the correct typmod. http://git.postgresql.org/pg/commitdiff/7f3014dce56c7975113809f2ff5e92cf7c1563a3
  • Avoid unused-variable warning in non-assert builds. Oversight in my commit b9896198cfbc1b0cd0c631d2af72ffe34bd4c7e5. http://git.postgresql.org/pg/commitdiff/a5c29d37aab00e9e70e72c97f2be29030f6ee84c
  • Remove comment claiming that PARAM_EXTERN Params always have typmod -1. This hasn't been true in quite some time, cf plpgsql's make_datum_param(). http://git.postgresql.org/pg/commitdiff/3200b15b20d9248be1b0f436ee787b2077d00298
  • Rethink function argument sorting in pg_dump. Commit 7b583b20b1c95acb621c71251150beef958bb603 created an unnecessary dump failure hazard by applying pg_get_function_identity_arguments() to every function in the database, even those that won't get dumped. This could result in snapshot-related problems if concurrent sessions are, for example, creating and dropping temporary functions, as noted by Marko Tiikkaja in bug #12832. While this is by no means pg_dump's only such issue with concurrent DDL, it's unfortunate that we added a new failure mode for cases that used to work, and even more so that the failure was created for basically cosmetic reasons (ie, to sort overloaded functions more deterministically). To fix, revert that patch and instead sort function arguments using information that pg_dump has available anyway, namely the names of the argument types. This will produce a slightly different sort ordering for overloaded functions than the previous coding; but applying strcmp directly to the output of pg_get_function_identity_arguments really was a bit odd anyway. The sorting will still be name-based and hence independent of possibly-installation-specific OID assignments. A small additional benefit is that sorting now works regardless of server version. Back-patch to 9.3, where the previous commit appeared. http://git.postgresql.org/pg/commitdiff/e3bfe6d84d4919433d8323cfb8194ca60d99f2c4
  • Fix erroneous error message for REINDEX SYSTEM. Missed case in commit fe263d115a7dd16095b8b8f1e943aff2bb4574d2. Sawada Masahiko http://git.postgresql.org/pg/commitdiff/ac0914285ac90bd411730c3219f226bbbbc57f3a
  • Code cleanup for REINDEX DATABASE/SCHEMA/SYSTEM. Fix some minor infelicities. Some of these things were introduced in commit fe263d115a7dd16095b8b8f1e943aff2bb4574d2, and some are older. http://git.postgresql.org/pg/commitdiff/90c35a9ed06c1353a0d3818c259e629ff09dba18
  • Fix documentation for libpq's PQfn(). The SGML docs claimed that 1-byte integers could be sent or received with the "isint" options, but no such behavior has ever been implemented in pqGetInt() or pqPutInt(). The in-code documentation header for PQfn() was even less in tune with reality, and the code itself used parameter names matching neither the SGML docs nor its libpq-fe.h declaration. Do a bit of additional wordsmithing on the SGML docs while at it. Since the business about 1-byte integers is a clear documentation bug, back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/1a0bc4c2bfc278b63965486b1525ad04a1f85989
  • Remove struct PQArgBlock from server-side header libpq/libpq.h. This struct is purely a client-side artifact. Perhaps there was once reason for the server to know it, but any such reason is lost in the mists of time. We certainly don't need two independent declarations of it. http://git.postgresql.org/pg/commitdiff/01cca2c1b1a0d52c83f250c50942ee00e62637ca
  • Cast to (void *) rather than (int *) when passing int64's to PQfn(). This is a possibly-vain effort to silence a Coverity warning about bogus endianness dependency. The code's fine, because it takes care of endianness issues for itself, but Coverity sees an int64 being passed to an int* argument and not unreasonably suspects something's wrong. I'm not sure if putting the void* cast in the way will shut it up; but it can't hurt and seems better from a documentation standpoint anyway, since the pointer is not used as an int* in this code path. Just for a bit of additional safety, verify that the result length is 8 bytes as expected. Back-patch to 9.3 where the code in question was added. http://git.postgresql.org/pg/commitdiff/ef75508efc789c79c5a5d4acd7ad5da85f1e4f08

Álvaro Herrera a poussé :

  • Add comment for "is_internal" parameter. This was missed in my commit f4c4335 of 9.3 vintage, so backpatch to that. http://git.postgresql.org/pg/commitdiff/6f9d79904748c26a58991942dc6719db558f77b0
  • Change many routines to return ObjectAddress rather than OID. The changed routines are mostly those that can be directly called by ProcessUtilitySlow; the intention is to make the affected object information more precise, in support for future event trigger changes. Originally it was envisioned that the OID of the affected object would be enough, and in most cases that is correct, but upon actually implementing the event trigger changes it turned out that ObjectAddress is more widely useful. Additionally, some command execution routines grew an output argument that's an object address which provides further info about the executed command. To wit: * for ALTER DOMAIN / ADD CONSTRAINT, it corresponds to the address of the new constraint * for ALTER OBJECT / SET SCHEMA, it corresponds to the address of the schema that originally contained the object. * for ALTER EXTENSION {ADD, DROP} OBJECT, it corresponds to the address of the object added to or dropped from the extension. There's no user-visible change in this commit, and no functional change either. Discussion: 20150218213255.GC6717@tamriel.snowman.net Reviewed-By: Stephen Frost, Andres Freund http://git.postgresql.org/pg/commitdiff/a2e35b53c39b2a27d3e332dc7c506539c306fd44
  • Silence warning in non-assert-enabled build. An OID return value was being used only for a (rather pointless) assert. Silence by removing the variable and the assert. Per note from Peter Geoghegan http://git.postgresql.org/pg/commitdiff/bf22d2707a2f47a7cc4caa239a14f2bf0a72bfd0
  • Fix user mapping object description. We were using "user mapping for user XYZ" as description for user mappings, but that's ambiguous because users can have mappings on multiple foreign servers; therefore change it to "for user XYZ on server UVW" instead. Object identities for user mappings are also updated in the same way, in branches 9.3 and above. The incomplete description string was introduced together with the whole SQL/MED infrastructure by commit cae565e503 of 8.4 era, so backpatch all the way back. http://git.postgresql.org/pg/commitdiff/cf34e373fcf42239a73f36e3054d9e9fbdc1e0de
  • Fix contrib/file_fdw's expected file. I forgot to update it on yesterday's cf34e373fcf. http://git.postgresql.org/pg/commitdiff/c6ee39bc8587042f018979ddd6ed9825acbbd3d8
  • Add some more tests on event triggers. Fabien Coelho. Reviewed by Robert Haas http://git.postgresql.org/pg/commitdiff/6510c832bbf91d52541c7aeefa371123abc2d832

Fujii Masao a poussé :

Peter Eisentraut a poussé :

Noah Misch a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Michael Paquier sent in a patch to fix broken Install.bat when target directory contains a space.
  • Jan de Visser sent in two more revisions of a patch to let pg ctl check the result of a postmaster config reload.
  • Gregory Stark and Haribabu Kommi traded patches to provide a catalog view of pg_hba.conf.
  • Joshua Berkus sent in four revisions of a patch to raise default for max_wal_segments to 1GB.
  • SAWADA Masahiko sent in another revision of a patch to implement REINDEX...VERBOSE.
  • Dean Rasheed sent in two more revisions of a patch to update RLS timings.
  • Álvaro Herrera sent in four more revisions of a patch to implement ALTER USER/ROLE ... CURRENT USER.
  • Michael Paquier sent in two more revisions of a patch to improve test coverage with pg_dump.
  • Shigeru HANADA and Ashutosh Bapat traded patches to implement push-down JOIN support for foreign tables.
  • Marko Kreen sent in a patch to fix excessive float lossiness in PL/Python.
  • Julien Tachoires sent in another revision of a patch to allow toast tables to be moved to a different tablespace.
  • Amit Kapila sent in another revision of a patch to implement parallel seq scan.
  • Kaigai Kouhei and Shigeru HANADA traded patches to add custom foreign join APIs.
  • Peter Geoghegan sent in another revision of a patch to add logical decoding support for ON CONFLICT UPDATE.
  • Peter Geoghegan sent in a patch to remove an obsolete SnapshotNow reference within snapbuild.c.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add regrole and regnamespace.
  • Michael Paquier sent in a patch to compare primary/HS standby in tests.
  • Robert Haas sent in a patch to make some fixes in tuplesort.
  • Rahila Syed sent in two more revisions of a patch to allow compressing improve full-page writes.
  • Fabien COELHO sent in four more revisions of a patch to improve pgbench syntax error messages.
  • David Rowley sent in another revision of a patch to improve performance for joins where outer side is unique.
  • Michael Paquier sent in a patch to install shared libraries in bin/ and lib/ on MSVC.
  • Michael Paquier sent in a patch to add dummy pstate fixes.
  • Peter Geoghegan sent in another revision of a patch to add INSERT ... ON CONFLICT IGNORE (and UPDATE).
  • Andreas Karlsson sent in another revision of a patch to use 128-bit integers for sum, avg and statistics aggregates.
  • Etsuro Fujita sent in another revision of a patch to make updating foreign tables in the Postgres FDW work faster.
  • SAWADA Masahiko sent in another revision of a patch to add a way to see the contents of configuration files via SQL.
  • Michael Paquier sent in two more revisions of a patch to add a table level log_autovacuum_min_duration.
  • Michael Paquier sent in a flock of patches to move the freeze parameters of VacuumStmt into a separate spot, eliminate VacuumStmt from lower level routines of ANALYZE and VACUUM, and add wraparound control parameters in VacuumStmt.
  • Bruce Momjian sent in two more revisions of a patch to help fix pg_upgrade with reference to rsync.
  • Robert Haas sent in another revision of a patch to implement parallel mode and parallel contexts.
  • Tom Lane sent in a patch to fix some weirdly pesimistic estimates in optimizer.
  • Álvaro Herrera sent in a flock of patches to patches add get_object_address support for user mappings, default ACLs, and operators and functions of operator families
  • Kyotaro HORIGUCHI and Tom Lane traded patches to clamp row number of join product by the row number calculated from joining paths.
  • Stephen Frost sent in another revision of a patch to add catalog_function_acls.
  • Marco Nenciarini sent in another revision of a patch to implement file-based incremental backup.
  • SAWADA Masahiko sent in a patch to fix an incorrect error message in REINDEX.
  • Fabrízio de Royes Mello sent in two revisions of a patch to fix an odd debug in walreceiver.
  • Peter Eisentraut sent in another revision of a patch to speed up make check-world.
  • Peter Eisentraut sent in another revision of a patch to add TRANSFORMS.
  • Pavel Stehule sent in a PoC patch to enforce casting to most common type automatically.
  • Dmitry Voronin sent in a patch to adds functions to sslinfo extension module: ssl_extension_names(), ssl_extension_value(text), and ssl_extension_is_critical(text).
  • Tomas Vondra sent in a patch to allow merging pgbench logs.
  • Tomas Vondra sent in a patch to allow logging both aggregate and transaction info in pgbench, rather having to choose one or the other.
  • Tom Lane sent in a patch to rethink the parameter access hooks for plpgsql's benefit.

par N Bougain le mardi 10 mars 2015 à 22h42

jeudi 5 mars 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 1er mars 2015

Le PUG de Manchester (Royaume-Uni) redémarre le 4 mars. Détails et RSVP ci-après : http://www.meetup.com/The-Manchester-PostgreSQL-Meetup/

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Heikki Linnakangas a poussé :

  • Fix potential deadlock with libpq non-blocking mode. If libpq output buffer is full, pqSendSome() function tries to drain any incoming data. This avoids deadlock, if the server e.g. sends a lot of NOTICE messages, and blocks until we read them. However, pqSendSome() only did that in blocking mode. In non-blocking mode, the deadlock could still happen. To fix, take a two-pronged approach: 1. Change the documentation to instruct that when PQflush() returns 1, you should wait for both read- and write-ready, and call PQconsumeInput() if it becomes read-ready. That fixes the deadlock, but applications are not going to change overnight. 2. In pqSendSome(), drain the input buffer before returning 1. This alleviates the problem for applications that only wait for write-ready. In particular, a slow but steady stream of NOTICE messages during COPY FROM STDIN will no longer cause a deadlock. The risk remains that the server attempts to send a large burst of data and fills its output buffer, and at the same time the client also sends enough data to fill its output buffer. The application will deadlock if it goes to sleep, waiting for the socket to become write-ready, before the server's data arrives. In practice, NOTICE messages and such that the server might be sending are usually short, so it's highly unlikely that the server would fill its output buffer so quickly. Backpatch to all supported versions. http://git.postgresql.org/pg/commitdiff/2a3f6e368babdac7b586a7d43105af60fc08b1a3
  • Refactor unit conversions code in guc.c. Replace the if-switch-case constructs with two conversion tables, containing all the supported conversions between human-readable unit strings and the base units used in GUC variables. This makes the code easier to read, and makes adding new units simpler. http://git.postgresql.org/pg/commitdiff/1b6302647302f459fdb63008c3842a1b0d43d1b7
  • Renumber GUC_* constants. This moves all the regular flags back together (for aesthetic reasons), and makes room for more GUC_UNIT_* types. http://git.postgresql.org/pg/commitdiff/0fec000365c25fd89ea583673de226e816dba60f
  • Replace checkpoint_segments with min_wal_size and max_wal_size. Instead of having a single knob (checkpoint_segments) that both triggers checkpoints, and determines how many checkpoints to recycle, they are now separate concerns. There is still an internal variable called CheckpointSegments, which triggers checkpoints. But it no longer determines how many segments to recycle at a checkpoint. That is now auto-tuned by keeping a moving average of the distance between checkpoints (in bytes), and trying to keep that many segments in reserve. The advantage of this is that you can set max_wal_size very high, but the system won't actually consume that much space if there isn't any need for it. The min_wal_size sets a floor for that; you can effectively disable the auto-tuning behavior by setting min_wal_size equal to max_wal_size. The max_wal_size setting is now the actual target size of WAL at which a new checkpoint is triggered, instead of the distance between checkpoints. Previously, you could calculate the actual WAL usage with the formula "(2 + checkpoint_completion_target) * checkpoint_segments + 1". With this patch, you set the desired WAL usage with max_wal_size, and the system calculates the appropriate CheckpointSegments with the reverse of that formula. That's a lot more intuitive for administrators to set. Reviewed by Amit Kapila and Venkata Balaji N. http://git.postgresql.org/pg/commitdiff/88e982302684246e8af785e78a467ac37c76dee9
  • Fix typo in README. Kyotaro Horiguchi http://git.postgresql.org/pg/commitdiff/dd58c6098f2f2fcea71c7125f9594268a24a38ad
  • Fix recovery_command -> restore_command typo in 8.3 release notes. Kyotaro Horiguchi http://git.postgresql.org/pg/commitdiff/347c74320d10bee458d1fec159aeda7143d31bfb

Fujii Masao a poussé :

  • Add GUC to control the time to wait before retrieving WAL after failed attempt. Previously when the standby server failed to retrieve WAL files from any sources (i.e., streaming replication, local pg_xlog directory or WAL archive), it always waited for five seconds (hard-coded) before the next attempt. For example, this is problematic in warm-standby because restore_command can fail every five seconds even while new WAL file is expected to be unavailable for a long time and flood the log files with its error messages. This commit adds new parameter, wal_retrieve_retry_interval, to control that wait time. Alexey Vasiliev and Michael Paquier, reviewed by Andres Freund and me. http://git.postgresql.org/pg/commitdiff/5d2b45e3f78a85639f30431181c06d4c3221c5a1
  • Add note about how to make the SRF detoasted arguments live accross calls. Andrew Gierth and Ali Akbar http://git.postgresql.org/pg/commitdiff/a7920b872fff36668a2d33157609024b851b5c2e

Andres Freund a poussé :

  • Guard against spurious signals in LockBufferForCleanup. When LockBufferForCleanup() has to wait for getting a cleanup lock on a buffer it does so by setting a flag in the buffer header and then wait for other backends to signal it using ProcWaitForSignal(). Unfortunately LockBufferForCleanup() missed that ProcWaitForSignal() can return for other reasons than the signal it is hoping for. If such a spurious signal arrives the wait flags on the buffer header will still be set. That then triggers "ERROR: multiple backends attempting to wait for pincount 1". The fix is simple, unset the flag if still set when retrying. That implies an additional spinlock acquisition/release, but that's unlikely to matter given the cost of waiting for a cleanup lock. Alternatively it'd have been possible to move responsibility for maintaining the relevant flag to the waiter all together, but that might have had negative consequences due to possible floods of signals. Besides being more invasive. This looks to be a very longstanding bug. The relevant code in LockBufferForCleanup() hasn't changed materially since its introduction and ProcWaitForSignal() was documented to return for unrelated reasons since 8.2. The master only patch series removing ImmediateInterruptOK made it much easier to hit though, as ProcSendSignal/ProcWaitForSignal now uses a latch shared with other tasks. Per discussion with Kevin Grittner, Tom Lane and me. Backpatch to all supported branches. Discussion: 11553.1423805224@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/bc208a5a2f9eb34bf795374fa5381e0c82073255
  • Reconsider when to wait for WAL flushes/syncrep during commit. Up to now RecordTransactionCommit() waited for WAL to be flushed (if synchronous_commit != off) and to be synchronously replicated (if enabled), even if a transaction did not have a xid assigned. The primary reason for that is that sequence's nextval() did not assign a xid, but are worthwhile to wait for on commit. This can be problematic because sometimes read only transactions do write WAL, e.g. HOT page prune records. That then could lead to read only transactions having to wait during commit. Not something people expect in a read only transaction. This lead to such strange symptoms as backends being seemingly stuck during connection establishment when all synchronous replicas are down. Especially annoying when said stuck connection is the standby trying to reconnect to allow syncrep again... This behavior also is involved in a rather complicated <= 9.4 bug where the transaction started by catchup interrupt processing waited for syncrep using latches, but didn't get the wakeup because it was already running inside the same overloaded signal handler. Fix the issue here doesn't properly solve that issue, merely papers over the problems. In 9.5 catchup interrupts aren't processed out of signal handlers anymore. To fix all this, make nextval() acquire a top level xid, and only wait for transaction commit if a transaction both acquired a xid and emitted WAL records. If only a xid has been assigned we don't uselessly want to wait just because of writes to temporary/unlogged tables; if only WAL has been written we don't want to wait just because of HOT prunes. The xid assignment in nextval() is unlikely to cause overhead in real-world workloads. For one it only happens SEQ_LOG_VALS/32 values anyway, for another only usage of nextval() without using the result in an insert or similar is affected. Discussion: 20150223165359.GF30784@awork2.anarazel.de, 369698E947874884A77849D8FE3680C2@maumau, 5CF4ABBA67674088B3941894E22A0D25@maumau Per complaint from maumau and Thom Brown Backpatch all the way back; 9.0 doesn't have syncrep, but it seems better to be consistent behavior across all maintained branches. http://git.postgresql.org/pg/commitdiff/fd6a3f3ad4067f1b8fc28e9de6e99e5936d82161

Tom Lane a poussé :

  • Further tweaking of raw grammar output to distinguish different inputs. Use a different A_Expr_Kind for LIKE/ILIKE/SIMILAR TO constructs, so that they can be distinguished from direct invocation of the underlying operators. Also, postpone selection of the operator name when transforming "x IN (select)" to "x = ANY (select)", so that those syntaxes can be told apart at parse analysis time. I had originally thought I'd also have to do something special for the syntaxes IS NOT DISTINCT FROM, IS NOT DOCUMENT, and x NOT IN (SELECT...), which the grammar translates as though they were NOT (construct). On reflection though, we can distinguish those cases reliably by noting whether the parse location shown for the NOT is the same as for its child node. This only requires tweaking the parse locations for NOT IN, which I've done here. These changes should have no effect outside the parser; they're just in support of being able to give accurate warnings for planned operator precedence changes. http://git.postgresql.org/pg/commitdiff/56be925e4b8f5b1c0e6716ca5cbe0360d1229f50
  • Improve parser's one-extra-token lookahead mechanism. There are a couple of places in our grammar that fail to be strict LALR(1), by requiring more than a single token of lookahead to decide what to do. Up to now we've dealt with that by using a filter between the lexer and parser that merges adjacent tokens into one in the places where two tokens of lookahead are necessary. But that creates a number of user-visible anomalies, for instance that you can't name a CTE "ordinality" because "WITH ordinality AS ..." triggers folding of WITH and ORDINALITY into one token. I realized that there's a better way. In this patch, we still do the lookahead basically as before, but we never merge the second token into the first; we replace just the first token by a special lookahead symbol when one of the lookahead pairs is seen. This requires a couple extra productions in the grammar, but it involves fewer special tokens, so that the grammar tables come out a bit smaller than before. The filter logic is no slower than before, perhaps a bit faster. I also fixed the filter logic so that when backing up after a lookahead, the current token's terminator is correctly restored; this eliminates some weird behavior in error message issuance, as is shown by the one change in existing regression test outputs. I believe that this patch entirely eliminates odd behaviors caused by lookahead for WITH. It doesn't really improve the situation for NULLS followed by FIRST/LAST unfortunately: those sequences still act like a reserved word, even though there are cases where they should be seen as two ordinary identifiers, eg "SELECT nulls first FROM ...". I experimented with additional grammar hacks but couldn't find any simple solution for that. Still, this is better than before, and it seems much more likely that we *could* somehow solve the NULLS case on the basis of this filter behavior than the previous one. http://git.postgresql.org/pg/commitdiff/d809fd0008a2e26de463f47b7aba0365264078f3
  • Fix dumping of views that are just VALUES(...) but have column aliases. The "simple" path for printing VALUES clauses doesn't work if we need to attach nondefault column aliases, because there's noplace to do that in the minimal VALUES() syntax. So modify get_simple_values_rte() to detect nondefault aliases and treat that as a non-simple case. This further exposes that the "non-simple" path never actually worked; it didn't produce valid syntax. Fix that too. Per bug #12789 from Curtis McEnroe, and analysis by Andrew Gierth. Back-patch to all supported branches. Before 9.3, this also requires back-patching the part of commit 092d7ded29f36b0539046b23b81b9f0bf2d637f1 that created get_simple_values_rte() to begin with; inserting the extra test into the old factorization of that logic would've been too messy. http://git.postgresql.org/pg/commitdiff/e9f1c01b71dcd11c86fc8516c06dae2e784b96fd
  • Fix over-optimistic caching in fetch_array_arg_replace_nulls(). When I rewrote this in commit 56a79a869bedc4bf6c35853642694cc0b0594dd2, I forgot that it's possible for the input array type to change from one call to the next (this can happen when applying the function to pg_statistic columns, for instance). Fix that. http://git.postgresql.org/pg/commitdiff/77903ede08845e55bd2a6c99b52d8da6926d6e84
  • Redefine MemoryContextReset() as deleting, not resetting, child contexts. That is, MemoryContextReset() now means what was formerly meant by MemoryContextResetAndDeleteChildren(), and the latter is now just a macro alias for the former. If you really want the functionality that was formerly provided by MemoryContextReset(), what you have to do is MemoryContextResetChildren() plus MemoryContextResetOnly() (which is a new API to reset *only* the named context and not touch its children). The reason for this change is that near fifteen years of experience has proven that there is noplace where old-style MemoryContextReset() is actually what you want. Making that the default behavior has led to lots of context-leakage bugs, while we've not found anyplace where it's actually necessary to keep the child contexts; at least the standard regression tests do not reveal anyplace where this change breaks anything. And there are upcoming patches that will introduce additional reasons why child contexts need to be removed. We could change existing calls of MemoryContextResetAndDeleteChildren to be just MemoryContextReset, but for the moment I'll leave them alone; they're not costing anything. http://git.postgresql.org/pg/commitdiff/eaa5808e8ec4e82ce1a87103a6b6f687666e4e4c
  • Suppress uninitialized-variable warning from less-bright compilers. The type variable must get set on first iteration of the while loop, but there are reasonably modern gcc versions that don't realize that. Initialize it with a dummy value. This undoes a removal of initialization in commit 654809e770ce270c0bb9de726c5df1ab193d60f0. http://git.postgresql.org/pg/commitdiff/d61f1a93271b859938b6cdb301a367cdb8abf81c
  • Improve mmgr README. Add documentation about the new reset callback mechanism. Also, at long last, recast the existing text so that it describes the current context mechanisms as established fact rather than something we're going to implement. Shoulda done that in 2001 or so ... http://git.postgresql.org/pg/commitdiff/c4f4c7ca99169ac609df67c2d0eb78162484420c
  • Fix planning of star-schema-style queries. Part of the intent of the parameterized-path mechanism was to handle star-schema queries efficiently, but some overly-restrictive search limiting logic added in commit e2fa76d80ba571d4de8992de6386536867250474 prevented such cases from working as desired. Fix that and add a regression test about it. Per gripe from Marc Cousin. This is arguably a bug rather than a new feature, so back-patch to 9.2 where parameterized paths were introduced. http://git.postgresql.org/pg/commitdiff/b514a7460d9127ddda6598307272c701cbb133b7
  • Track typmods in plpgsql expression evaluation and assignment. The main value of this change is to avoid expensive I/O conversions when assigning to a variable that has a typmod specification, if the value to be assigned is already known to have the right typmod. This is particularly valuable for arrays with typmod specifications; formerly, in an assignment to an array element the entire array would invariably get put through double I/O conversion to check the typmod, to absolutely no purpose since we'd already properly coerced the new element value. Extracted from my "expanded arrays" patch; this seems worth committing separately, whatever becomes of that patch, since it's really an independent issue. As long as we're changing the function signatures, take the opportunity to rationalize the argument lists of exec_assign_value, exec_cast_value, and exec_simple_cast_value; that is, put the arguments into a saner order, and get rid of the bizarre choice to pass exec_assign_value's isNull flag by reference. http://git.postgresql.org/pg/commitdiff/e524cbdc45ec6d677b1dd49ee64dd403959eda0f
  • Invent a memory context reset/delete callback mechanism. This allows cleanup actions to be registered to be called just before a particular memory context's contents are flushed (either by deletion or MemoryContextReset). The patch in itself has no use-cases for this, but several likely reasons for wanting this exist. In passing, per discussion, rearrange some boolean fields in struct MemoryContextData so as to avoid wasted padding space. For safety, this requires making allowInCritSection's existence unconditional; but I think that's a better approach than what was there anyway. http://git.postgresql.org/pg/commitdiff/f65e8270587f3e9b8224e20f7d020ed1f816dfe1
  • Move memory context callback declarations into palloc.h. Initial experience with this feature suggests that instances of MemoryContextCallback are likely to propagate into some widely-used headers over time. As things stood, that would result in pulling memutils.h or at least memnodes.h into common headers, which does not seem desirable. Instead, let's decide that this feature is part of the "ordinary palloc user" API rather than the "specialized context management" API, and as such should be declared in palloc.h not memutils.h. http://git.postgresql.org/pg/commitdiff/097fe194aa7c590b4fa43d5e40c083940859c286
  • Use the typcache to cache constraints for domain types. Previously, we cached domain constraints for the life of a query, or really for the life of the FmgrInfo struct that was used to invoke domain_in() or domain_check(). But plpgsql (and probably other places) are set up to cache such FmgrInfos for the whole lifespan of a session, which meant they could be enforcing really stale sets of constraints. On the other hand, searching pg_constraint once per query gets kind of expensive too: testing says that as much as half the runtime of a trivial query such as "SELECT 0::domaintype" went into that. To fix this, delegate the responsibility for tracking a domain's constraints to the typcache, which has the infrastructure needed to detect syscache invalidation events that signal possible changes. This not only removes unnecessary repeat reads of pg_constraint, but ensures that we never apply stale constraint data: whatever we use is the current data according to syscache rules. Unfortunately, the current configuration of the system catalogs means we have to flush cached domain-constraint data whenever either pg_type or pg_constraint changes, which happens rather a lot (eg, creation or deletion of a temp table will do it). It might be worth rearranging things to split pg_constraint into two catalogs, of which the domain constraint one would probably be very low-traffic. That's a job for another patch though, and in any case this patch should improve matters materially even with that handicap. This patch makes use of the recently-added memory context reset callback feature to manage the lifespan of domain constraint caches, so that we don't risk deleting a cache that might be in the midst of evaluation. Although this is a bug fix as well as a performance improvement, no back-patch. There haven't been many if any field complaints about stale domain constraint checks, so it doesn't seem worth taking the risk of modifying data structures as basic as MemoryContexts in back branches. http://git.postgresql.org/pg/commitdiff/8abb3cda0ddc00a0ab98977a1633a95b97068d4e

Álvaro Herrera a poussé :

  • Fix stupid merge errors in previous commit. Brown paper bag installed permanently. http://git.postgresql.org/pg/commitdiff/d1712d01d01d2c14355fad497fa7a6ae6e33694f
  • Support more commands in event triggers. COMMENT, SECURITY LABEL, and GRANT/REVOKE now also fire ddl_command_start and ddl_command_end event triggers, when they operate on database-local objects. Reviewed-By: Michael Paquier, Andres Freund, Stephen Frost http://git.postgresql.org/pg/commitdiff/296f3a6053844089bc533630fffafaba8f016384
  • Fix table_rewrite event trigger for ALTER TYPE/SET DATA TYPE CASCADE. When a composite type being used in a typed table is modified by way of ALTER TYPE, a table rewrite occurs appearing to come from ALTER TYPE. The existing event_trigger.c code was unable to cope with that and raised a spurious error. The fix is just to accept that command tag for the event, and document this properly. Noted while fooling with deparsing of DDL commands. This appears to be an oversight in commit 618c9430a. Thanks to Mark Wong for documentation wording help. http://git.postgresql.org/pg/commitdiff/3f190f67eb45ae61d696fbce8ab48d246a12f709
  • Make CREATE OR REPLACE VIEW internally more consistent. The way that columns are added to a view is by calling AlterTableInternal with special subtype AT_AddColumnToView; but that subtype is changed to AT_AddColumnRecurse by ATPrepAddColumn. This has no visible effect in the current code, since views cannot have inheritance children (thus the recursion step is a no-op) and adding a column to a view is executed identically to doing it to a table; but it does make a difference for future event trigger code keeping track of commands, because the current situation leads to confusing the case with a normal ALTER TABLE ADD COLUMN. Fix the problem by passing a flag to ATPrepAddColumn to prevent it from changing the command subtype. The event trigger code can then properly ignore the subcommand. (We could remove the call to ATPrepAddColumn, since views are never typed, and there is never a need for recursion, which are the two conditions that are checked by ATPrepAddColumn; but it seems more future-proof to keep the call in place.) http://git.postgresql.org/pg/commitdiff/fbef4342a86522c98cd605891ad8c1145a61d191
  • Fix a couple of trivial issues in jsonb.c. Typo "aggreagate" appeared three times, and the return value of function JsonbIteratorNext() was being assigned to an int variable in a bunch of places. http://git.postgresql.org/pg/commitdiff/654809e770ce270c0bb9de726c5df1ab193d60f0
  • Fix intermittent failure in event_trigger test. As evidenced by measles in buildfarm. Pointed out by Tom Lane. http://git.postgresql.org/pg/commitdiff/e059e02e43cd825616192d010e9e638a96ad4717

Peter Eisentraut a poussé :

  • Fix invalid DocBook XML http://git.postgresql.org/pg/commitdiff/b007bee1f6962ad1007056f64b9eb2e505fa6806
  • Error when creating names too long for tar format The tar format (at least the version we are using), does not support file names or symlink targets longer than 99 bytes. Until now, the tar creation code would silently truncate any names that are too long. (Its original application was pg_dump, where this never happens.) This creates problems when running base backups over the replication protocol. The most important problem is when a tablespace path is longer than 99 bytes, which will result in a truncated tablespace path being backed up. Less importantly, the basebackup protocol also promises to back up any other files it happens to find in the data directory, which would also lead to file name truncation if someone put a file with a long name in there. Now both of these cases result in an error during the backup. Add tests that fail when a too-long file name or symlink is attempted to be backed up. Reviewed-by: Robert Haas <robertmhaas@gmail.com> http://git.postgresql.org/pg/commitdiff/23a78352c0a0dc21d6120bd868f0b2d07395b537

Michael Meskes a poussé :

Stephen Frost a poussé :

  • Add locking clause for SB views for update/delete. In expand_security_qual(), we were handling locking correctly when a PlanRowMark existed, but not when we were working with the target relation (which doesn't have any PlanRowMarks, but the subquery created for the security barrier quals still needs to lock the rows under it). Noted by Etsuro Fujita when working with the Postgres FDW, which wasn't properly issuing a SELECT ... FOR UPDATE to the remote side under a DELETE. Back-patch to 9.4 where updatable security barrier views were introduced. Per discussion with Etsuro Fujita and Dean Rasheed. http://git.postgresql.org/pg/commitdiff/6f9bd50eabb0a4960e94c83dac8855771c9f340d
  • Add hasRowSecurity to copyfuncs/outfuncs. The RLS patch added a hasRowSecurity field to PlannerGlobal and PlannedStmt but didn't update nodes/copyfuncs.c and nodes/outfuncs.c to reflect those additional fields. Correct that by adding entries to the appropriate functions for those fields. Pointed out by Robert Haas. http://git.postgresql.org/pg/commitdiff/62a4a1af5d1d84f0023bc3816c204191b0f4f49f
  • Fix targetRelation initializiation in prepsecurity. In 6f9bd50eabb0a4960e94c83dac8855771c9f340d, we modified expand_security_quals() to tell expand_security_qual() about when the current RTE was the targetRelation. Unfortunately, that commit initialized the targetRelation variable used outside of the loop over the RTEs instead of at the start of it. This patch moves the variable and the initialization of it into the loop, where it should have been to begin with. Pointed out by Dean Rasheed. Back-patch to 9.4 as the original commit was. http://git.postgresql.org/pg/commitdiff/ee4ddcb38a0abfdb8f7302bbc332a1cb92888ed1

Noah Misch a poussé :

  • Free SQLSTATE and SQLERRM no earlier than other PL/pgSQL variables. "RETURN SQLERRM" prompted plpgsql_exec_function() to read from freed memory. Back-patch to 9.0 (all supported versions). Little code ran between the premature free and the read, so non-assert builds are unlikely to witness user-visible consequences. http://git.postgresql.org/pg/commitdiff/f5ef00aed4c39645716cabb2e4cf1ef3598fcde7
  • Unlink static libraries before rebuilding them. When the library already exists in the build directory, "ar" preserves members not named on its command line. This mattered when, for example, a "configure" rerun dropped a file from $(LIBOBJS). libpgport carried the obsolete member until "make clean". Back-patch to 9.0 (all supported versions). http://git.postgresql.org/pg/commitdiff/424793fa5dc631254f69d5ee8d7d7d6de2976f60
  • Add transform functions for AT TIME ZONE. This makes "ALTER TABLE tabname ALTER tscol TYPE ... USING tscol AT TIME ZONE 'UTC'" skip rewriting the table when altering from "timestamp" to "timestamptz" or vice versa. While it would be nicer still to optimize this in the absence of the USING clause given timezone==UTC, transform functions must consult IMMUTABLE facts only. http://git.postgresql.org/pg/commitdiff/b8a18ad4850ea5ad7884aa6ab731fd392e73b4ad

Andrew Dunstan a poussé :

  • Render infinite date/timestamps as 'infinity' for json/jsonb. Commit ab14a73a6c raised an error in these cases and later the behaviour was copied to jsonb. This is what the XML code, which we then adopted, does, as the XSD types don't accept infinite values. However, json dates and timestamps are just strings as far as json is concerned, so there is no reason not to render these values as 'infinity'. The json portion of this is backpatched to 9.4 where the behaviour was introduced. The jsonb portion only affects the development branch. Per gripe on pgsql-general. http://git.postgresql.org/pg/commitdiff/bda76c1c8cfb1d11751ba6be88f0242850481733

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Corey Huinker sent in another revision of a patch to add polymorphic functions to dblink.
  • Tom Lane sent in two more revisions of a patch to make operator precedent standards compliant and sane.
  • Michael Paquier sent in two more revisions of a patch to enforce creation of input and output paths in pg_regress, and add regress_dynamic as a test module.
  • David Steele sent in two more revisions of a patch to add pg_audit as an extension.
  • Álvaro Herrera sent in a patch to remove the OBJECT_ATTRIBUTE symbol.
  • Peter Eisentraut sent in another revision of a patch to add logical decoding support for ON CONFLICT UPDATE.
  • Andres Freund sent in a patch to reconsider when to flush WAL and wait for syncrep while committing.
  • Rahila Syed sent in three more revisions of a patch to compress full-page writes.
  • Álvaro Herrera and David Fetter traded patches to fix a bug in psql's handling of connect strings and URIs in psql's \connect.
  • Tomas Vondra sent in another revision of a patch to separate logical from physical column ordering.
  • Michael Paquier and Stephen Frost traded patches to fix a dependency issue in pg_dump which mis-ordered tables in extensions.
  • Kyotaro HORIGUCHI sent in another revision of a patch to ensure that pg_basebackup sends feedback regularly, even under heavy loads.
  • Tom Lane sent in a patch to ensure that NOT LIKE, etc., have reasonable precedence.
  • Andreas Karlsson sent in a patch to fix some autoconf deprecation warnings.
  • Amit Langote sent in two revisions of a patch to implement native partitioning.
  • Álvaro Herrera sent in two flocks of patches for deparsing utility commands.
  • Alexander Korotkov sent in another revision of a patch to add KNN-GiST with recheck.
  • Michael Paquier sent in a patch to add NULL-pointer check and incorrect comment for pstate in addRangeTableEntry.
  • Fabien COELHO sent in two more revisions of a patch to add more tests on event triggers.
  • Tom Lane sent in another revision of a patch to manipulate complex types as non-contiguous structures in-memory.
  • Fabrízio de Royes Mello sent in a patch to add CINE for ALTER TABLE ... ADD COLUMN.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add new OID alias type regrole.
  • Tom Lane sent in a patch to make MemoryContextReset delete children.
  • Tom Lane sent in a patch to add MemoryContext reset/delete callbacks.
  • Haribabu Kommi sent in another revision of a patch to provide a catalog view to the pg_hba.conf file.
  • Kyotaro HORIGUCHI and Kevin Grittner traded patches to reduce pinning in btree indexes.
  • Corey Huinker sent in another revision of a patch to allow FETCH limited by bytes.
  • Anastasia Lubennikova and Heikki Linnakangas traded patches to add index-only scans for GiST.
  • Michael Paquier sent in a patch to rm static libraries before rebuilding.
  • Stephen Frost sent in another revision of a patch to add role attributes for various operations.
  • Tom Lane sent in a patch to cache domain constraints properly.
  • Michael Paquier sent in a patch to move freeze parameters of VacuumStmt into a separate structure.
  • Andrew Dunstan sent in another revision of a patch to implement mogrify and indent features for jsonb.

par N Bougain le jeudi 5 mars 2015 à 09h32

jeudi 26 février 2015

Adrien Nayrat

Backup, restauration – Part 6

J’ai consacré les précédents articles aux différents mécanismes de réplications. Le principe est à chaque fois le même, on utilise une copie d’une instance Postgres puis la réplication se fait par rejeu des journaux de transaction (soit par transfert de fichier ou par flux).

Cet article va aborder les différentes technique de sauvegarde … ainsi que la restauration. On ne pense pas assez à la restauration : “la priorité est la sauvegarde, la restauration on verra quand on aura un peu de temps”.

  1. Sauvegarde fichier d’une instance (manuelle + pg_base_backup)
  2. Sauvegarde par dump des bases avec pg_dump
  3. Restauration (manuelle + pg_restore)
  4. Restauration PITR

Sauvegarde fichier d’une instance

La sauvegarde la plus simple et la plus basique est la sauvegarde à froid des fichier d’une instance. Il suffit d’arrêter postgres et de faire une copie du dossier data de postgres. Cette solution a l’inconvénient majeur de devoir arrêter Postgres. Nous allons voir d’autres solutions plus élaborées qui évitent d’éteindre le moteur.

Sauvegarde par dump des bases avec pg_dump et pg_dumpall

Pour éviter d’éteindre le serveur on peut tout simplement faire un dump de la base de donnée avec pg_dump. Je ne vais pas recopier la page de documentation de pg_dump ici mais me contenter de vous présenter les options utiles de l’outil.

L’inconvénient ou l’avantage de cet outil est qu’il ne sauvegarde pas toutes les bases d’une instance, il faut donc scripter le tout pour qu’il sauvegarde base par base. J’ai écrit “inconvénient ou avantage”, en effet vous pouvez décider de ne restaurer qu’une seule base ou avoir des politique de sauvegarde différentes en fonction des bases. Si vous souhaitez sauvegarder toutes les bases d’un coup vous pouvez utiliser pg_dumpall. Attention, avec pg_dump vous ne sauvegardez pas les rôles donc en cas de restauration vous devriez recréer les rôles. Pour y remédier vous pouvez utiliser –roles-only.

Par défaut, pg_dump va sauvegarder les tables, les données de schéma. Il est possible de les sélectionner avec les options : –data-only ou –schema-only
Petite astuce, à partir de la version 9.3 pg_dump peut paralléliser la sauvegarde des tables avec l’option –jobs=njobs

Pour approfondir :
http://docs.postgresqlfr.org/9.3/app-pgdump.html

http://www.dalibo.org/_media/pgdayfr-postgresql_9.3-start.pdf

Restauration depuis un dump

pgsql

La technique la plus simple consiste à envoyer à psql les commandes issues du dump. Voici mon dump :

pg_dump -h /var/run/postgresql/ test

--
-- PostgreSQL database dump
--

SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET client_min_messages = warning;

--
-- Name: plpgsql; Type: EXTENSION; Schema: -; Owner:
--

CREATE EXTENSION IF NOT EXISTS plpgsql WITH SCHEMA pg_catalog;


--
-- Name: EXTENSION plpgsql; Type: COMMENT; Schema: -; Owner:
--

COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';


SET search_path = public, pg_catalog;

SET default_tablespace = '';

SET default_with_oids = false;

--
-- Name: films; Type: TABLE; Schema: public; Owner: postgres; Tablespace:
--

CREATE TABLE films (
    code character(5),
    title character varying(40),
    did integer,
    date_prod date,
    kind character varying(10),
    len interval hour to minute
);


ALTER TABLE public.films OWNER TO postgres;

--
-- Data for Name: films; Type: TABLE DATA; Schema: public; Owner: postgres
--

COPY films (code, title, did, date_prod, kind, len) FROM stdin;
bla     rambo   12      2014-12-01      action  01:00:00
\.


--
-- Name: production; Type: CONSTRAINT; Schema: public; Owner: postgres; Tablespace:
--

ALTER TABLE ONLY films
    ADD CONSTRAINT production UNIQUE (date_prod);


--
-- Name: public; Type: ACL; Schema: -; Owner: postgres
--

REVOKE ALL ON SCHEMA public FROM PUBLIC;
REVOKE ALL ON SCHEMA public FROM postgres;
GRANT ALL ON SCHEMA public TO postgres;
GRANT ALL ON SCHEMA public TO PUBLIC;


--
-- PostgreSQL database dump complete
--

Comme vous pouvez le constater, pg_dump utilise des “copy” plutôt que des “insert”. La commande copy est bien plus rapide pour l’import en masse de données.

Pour restaurer le dump :

psql -h /var/run/postgresql/ test -f dump.sql
SET
SET
SET
SET
SET
SET
CREATE EXTENSION
COMMENT
SET
SET
SET
CREATE TABLE
ALTER TABLE
ALTER TABLE
REVOKE
REVOKE
GRANT
GRANT

Pour approfondir :
Documentation de pg_dump
Documentation de pg_dumpall

pg_restore

Cette restauration est assez simple, postgres fourni l’utilitaire pg_restore qui offre des fonctionnalités intéressantes telles que :

  • Parallélisation
  • Sélection des sections à restaurer. Très utile si vous ne souhaitez restaurer qu’une seule table.

Pour pouvoir utiliser pg_restore il faut spécifier le format “custom” à pg_dump: “–format=custom” ou “-F c”. En effet si vous fournissez un fichier au format “à plat” vous aurez le message suivant :

pg_restore: [archiveur] Le fichier en entrée semble être une sauvegarde au format texte. Merci d'utiliser psql.

Je ne vais pas recopier la documentation de pg_restore mais juste vous fournir les options les plus intéressantes :

Parallélisation :

-j number-of-jobs
--jobs=number-of-jobs

Lister les sections du dump :

<tt class="OPTION">-l</tt>
 <tt class="OPTION">--list</tt>

Exemple :

;
; Archive created at Sun Jan 11 12:40:21 2015
;     dbname: test
;     TOC Entries: 11
;     Compression: -1
;     Dump Version: 1.12-0
;     Format: CUSTOM
;     Integer: 4 bytes
;     Offset: 8 bytes
;     Dumped from database version: 9.3.5
;     Dumped by pg_dump version: 9.3.5
;
;
; Selected TOC Entries:
;
1939; 1262 16486 DATABASE - test postgres
6; 2615 2200 SCHEMA - public postgres
1940; 0 0 COMMENT - SCHEMA public postgres
1941; 0 0 ACL - public postgres
171; 3079 11756 EXTENSION - plpgsql
1942; 0 0 COMMENT - EXTENSION plpgsql
170; 1259 16502 TABLE public films postgres
1934; 0 16502 TABLE DATA public films postgres
1826; 2606 16506 CONSTRAINT public production postgres

Si je souhaite supprimer quelques sections du dump il me suffit de commenter la ligne avec un point-virgule, de sauvegarder le fichier (db.list) puis de lancer pg_restore avec l’option “-L” :

pg_restore -L db.list db.dump

Pour approfondir : Documentation de pg_restore

Nous venons de voir les différentes technique pour restaurer des données. Cette façon de faire présente quelques inconvénients. Il faut faire des backups régulier. On peut avoir une perte de données conséquente depuis le dernier le backup.

En mettant en place une réplication on peut penser que le problème est réglé. Cependant la réplication nous protège d’une perte du maître mais pas d’une erreur humaine. Un DROP TABLE sur une mauvaise base par exemple! Ainsi le RPO correspond au temps qui s’est écoulé depuis le dernier backup.

 

Archivage des WAL + restauration PITR

A venir dans un prochain article.

Edit : Le voici : Restauration PITR – Part 7

 

 

FacebookTwitterGoogle+ViadeoPrintEmailGoogle GmailLinkedInPartager

par Adrien Nayrat le jeudi 26 février 2015 à 17h30

mardi 24 février 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 22 février 2015

Le PUG indien organise un PGday à Bengaluru (État du Karnataka en Inde) le 11 avril 2015. RSVP : http://www.meetup.com/India-PUG/events/220553997/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en février

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Fix null-pointer-deref crash while doing COPY IN with check constraints. In commit bf7ca15875988a88e97302e012d7c4808bef3ea9 I introduced an assumption that an RTE referenced by a whole-row Var must have a valid eref field. This is false for RTEs constructed by DoCopy, and there are other places taking similar shortcuts. Perhaps we should make all those places go through addRangeTableEntryForRelation or its siblings instead of having ad-hoc logic, but the most reliable fix seems to be to make the new code in ExecEvalWholeRowVar cope if there's no eref. We can reasonably assume that there's no need to insert column aliases if no aliases were provided. Add a regression test case covering this, and also verifying that a sane column name is in fact available in this situation. Although the known case only crashes in 9.4 and HEAD, it seems prudent to back-patch the code change to 9.2, since all the ingredients for a similar failure exist in the variant patch applied to 9.3 and 9.2. Per report from Jean-Pierre Pelletier. http://git.postgresql.org/pg/commitdiff/08361cea2b333189fa3736f2dc6452cef654923d
  • Rationalize the APIs of array element/slice access functions. The four functions array_ref, array_set, array_get_slice, array_set_slice have traditionally declared their array inputs and results as being of type "ArrayType *". This is a lie, and has been since Berkeley days, because they actually also support "fixed-length array" types such as "name" and "point"; not to mention that the inputs could be toasted. These values should be declared Datum instead to avoid confusion. The current coding already risks possible misoptimization by compilers, and it'll get worse when "expanded" array representations become a valid alternative. However, there's a fair amount of code using array_ref and array_set with arrays that *are* known to be ArrayType structures, and there might be more such places in third-party code. Rather than cluttering those call sites with PointerGetDatum/DatumGetArrayTypeP cruft, what I did was to rename the existing functions to array_get_element/array_set_element, fix their signatures, then reincarnate array_ref/array_set as backwards compatibility wrappers. array_get_slice/array_set_slice have no such constituency in the core code, and probably not in third-party code either, so I just changed their APIs. http://git.postgresql.org/pg/commitdiff/e983c4d1aa42d613542cf222e222b034918374b1
  • Use fast path in plpgsql's RETURN/RETURN NEXT in more cases. exec_stmt_return() and exec_stmt_return_next() have fast-path code for handling a simple variable reference (i.e. "return var") without going through the full expression evaluation machinery. For some reason, pl_gram.y was under the impression that this fast path only applied for record/row variables; but in reality code for handling regular scalar variables has been there all along. Adjusting the logic to allow that code to be used actually results in a net savings of code in pl_gram.y (by eliminating some redundancy), and it buys a measurable though not very impressive amount of speedup. Noted while fooling with my expanded-array patch, wherein this makes a much bigger difference because it enables returning an expanded array variable without an extra flattening step. But AFAICS this is a win regardless, so commit it separately. http://git.postgresql.org/pg/commitdiff/9e3ad1aac52454569393a947c06be0d301749362
  • Fix misuse of memcpy() in check_ip(). The previous coding copied garbage into a local variable, pretty much ensuring that the intended test of an IPv6 connection address against a promoted IPv4 address from pg_hba.conf would never match. The lack of field complaints likely indicates that nobody realized this was supposed to work, which is unsurprising considering that no user-facing docs suggest it should work. In principle this could have led to a SIGSEGV due to reading off the end of memory, but since the source address would have pointed to somewhere in the function's stack frame, that's quite unlikely. What led to discovery of the bug is Hugo Osvaldo Barrera's report of a crash after an OS upgrade, which is probably because he is now running a system in which memcpy raises abort() upon detecting overlapping source and destination areas. (You'd have to additionally suppose some things about the stack frame layout to arrive at this conclusion, but it seems plausible.) This has been broken since the code was added, in commit f3aec2c7f51904e7, so back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/cb66f495f5d0c204f051971f2c549d5c3ac850ea
  • Remove code to match IPv4 pg_hba.conf entries to IPv4-in-IPv6 addresses. In investigating yesterday's crash report from Hugo Osvaldo Barrera, I only looked back as far as commit f3aec2c7f51904e7 where the breakage occurred (which is why I thought the IPv4-in-IPv6 business was undocumented). But actually the logic dates back to commit 3c9bb8886df7d56a and was simply broken by erroneous refactoring in the later commit. A bit of archives excavation shows that we added the whole business in response to a report that some 2003-era Linux kernels would report IPv4 connections as having IPv4-in-IPv6 addresses. The fact that we've had no complaints since 9.0 seems to be sufficient confirmation that no modern kernels do that, so let's just rip it all out rather than trying to fix it. Do this in the back branches too, thus essentially deciding that our effective behavior since 9.0 is correct. If there are any platforms on which the kernel reports IPv4-in-IPv6 addresses as such, yesterday's fix would have made for a subtle and potentially security-sensitive change in the effective meaning of IPv4 pg_hba.conf entries, which does not seem like a good thing to do in minor releases. So let's let the post-9.0 behavior stand, and change the documentation to match it. In passing, I failed to resist the temptation to wordsmith the description of pg_hba.conf IPv4 and IPv6 address entries a bit. A lot of this text hasn't been touched since we were IPv4-only. http://git.postgresql.org/pg/commitdiff/2e105def09f27d49c1761abab06b427cfaa5d304
  • Fix EXPLAIN output for cases where parent table is excluded by constraints. The previous coding in EXPLAIN always labeled a ModifyTable node with the name of the target table affected by its first child plan. When originally written, this was necessarily the parent table of the inheritance tree, so everything was unconfusing. But when we added NO INHERIT constraints, it became possible for the parent table to be deleted from the plan by constraint exclusion while still leaving child tables present. This led to the ModifyTable plan node being labeled with the first surviving child, which was deemed confusing. Fix it by retaining the parent table's RT index in a new field in ModifyTable. Etsuro Fujita, reviewed by Ashutosh Bapat and myself http://git.postgresql.org/pg/commitdiff/abe45a9b315d7b3739f442597f570f9454bd466d
  • Fix failure to honor -Z compression level option in pg_dump -Fd. cfopen() and cfopen_write() failed to pass the compression level through to zlib, so that you always got the default compression level if you got any at all. In passing, also fix these and related functions so that the correct errno is reliably returned on failure; the original coding supposes that free() cannot change errno, which is untrue on at least some platforms. Per bug #12779 from Christoph Berg. Back-patch to 9.1 where the faulty code was introduced. Michael Paquier http://git.postgresql.org/pg/commitdiff/0e7e355f27302b62af3e1add93853ccd45678443
  • Fix placement of "SET row_security" command issuance in pg_dump. Somebody apparently threw darts at the code to decide where to insert these. They certainly didn't proceed by adding them where other similar SETs were handled. This at least broke pg_restore, and perhaps other use-cases too. http://git.postgresql.org/pg/commitdiff/297b2c1ef9878f84a9951beadf831ef390227238
  • Split array_push into separate array_append and array_prepend functions. There wasn't any good reason for a single C function to implement both these SQL functions: it saved very little code overall, and it required significant pushups to re-determine at runtime which case applied. Redoing it as two functions ends up with just slightly more lines of code, but it's simpler to understand, and faster too because we need not repeat syscache lookups on every call. An important side benefit is that this eliminates the only case in which different aliases of the same C function had both anyarray and anyelement arguments at the same position, which would almost always be a mistake. The opr_sanity regression test will now notice such mistakes since there's no longer a valid case where it happens. http://git.postgresql.org/pg/commitdiff/56a79a869bedc4bf6c35853642694cc0b0594dd2
  • Update assorted TOAST-related documentation. While working on documentation for expanded arrays, I noticed a number of details in the TOAST-related documentation that were already inaccurate or obsolete. This should be fixed independently of whether expanded arrays get in or not. One issue is that the already existing indirect-pointer facility was not documented at all. Also, the documentation says that you only need to use VARSIZE/SET_VARSIZE if you've made your variable-length type TOAST-aware, but actually we've forced that business on all varlena types even if they've opted out of TOAST by setting storage = plain. Wordsmith a few other things too, like an amusingly archaic claim that there are few 64-bit machines. I thought about back-patching this, but since all this doco is oriented to hackers and C-coded extension authors, fixing it in HEAD is probably good enough. http://git.postgresql.org/pg/commitdiff/9bb955c8286c20474b5462eea3e3cf76c694d88f
  • Add pg_stat_get_snapshot_timestamp() to show statistics snapshot timestamp. Per discussion, this could be useful for purposes such as programmatically detecting a nonresponding stats collector. We already have the timestamp anyway, it's just a matter of providing a SQL-accessible function to fetch it. Matt Kelly, reviewed by Jim Nasby http://git.postgresql.org/pg/commitdiff/2fb7a75f37d0beca80f45e15736ec8d50064228a
  • Use "#ifdef CATALOG_VARLEN" to protect nullable fields of pg_authid. This gives a stronger guarantee than a mere comment against accessing these fields as simple struct members. Since rolpassword is in fact varlena, it's not clear why these didn't get marked from the beginning, but let's do it now. Michael Paquier http://git.postgresql.org/pg/commitdiff/692bd09ad12a27fda3566787b089fbd394d7b65b
  • Remove unused variable. Per buildfarm. http://git.postgresql.org/pg/commitdiff/9aa53bbd15ef65a4be6c09709fbe985898ae9ad7
  • Some more FLEXIBLE_ARRAY_MEMBER hacking. http://git.postgresql.org/pg/commitdiff/5740be6d6e39dd85587aa71b3bd1fb0a423858b0
  • Use FLEXIBLE_ARRAY_MEMBER in struct varlena. This forces some minor coding adjustments in tuptoaster.c and inv_api.c, but the new coding there is cleaner anyway. Michael Paquier http://git.postgresql.org/pg/commitdiff/e38b1eb0986990d539e056a65c6b122b295ce932
  • Use FLEXIBLE_ARRAY_MEMBER in struct RecordIOData. I (tgl) fixed this last night in rowtypes.c, but I missed that the code had been copied into a couple of other places. Michael Paquier http://git.postgresql.org/pg/commitdiff/c110eff1324f5c882c737ad988191ed4a54c4936
  • Use FLEXIBLE_ARRAY_MEMBER in some more places. Fix a batch of structs that are only visible within individual .c files. Michael Paquier http://git.postgresql.org/pg/commitdiff/33a3b03d636b529b27f869e332b6344d52635331
  • Fix statically allocated struct with FLEXIBLE_ARRAY_MEMBER member. clang complains about this, not unreasonably, so define another struct that's explicitly for a WordEntryPos with exactly one element. While at it, get rid of pretty dubious use of a static variable for more than one purpose --- if it were being treated as const maybe I'd be okay with this, but it isn't. http://git.postgresql.org/pg/commitdiff/33b2a2c97f3dd4cf8bbc2c020e34129886367b72
  • Some more FLEXIBLE_ARRAY_MEMBER fixes. http://git.postgresql.org/pg/commitdiff/f2874feb7c4e71200539c0287544eeebd8fcb3d4
  • Use FLEXIBLE_ARRAY_MEMBER in a bunch more places. Replace some bogus "x[1]" declarations with "x[FLEXIBLE_ARRAY_MEMBER]". Aside from being more self-documenting, this should help prevent bogus warnings from static code analyzers and perhaps compiler misoptimizations. This patch is just a down payment on eliminating the whole problem, but it gets rid of a lot of easy-to-fix cases. Note that the main problem with doing this is that one must no longer rely on computing sizeof(the containing struct), since the result would be compiler-dependent. Instead use offsetof(struct, lastfield). Autoconf also warns against spelling that offsetof(struct, lastfield[0]). Michael Paquier, review and additional fixes by me. http://git.postgresql.org/pg/commitdiff/09d8d110a604e52216102e73fb8475b7aa88f1d1
  • Fix misparsing of empty value in conninfo_uri_parse_params(). After finding an "=" character, the pointer was advanced twice when it should only advance once. This is harmless as long as the value after "=" has at least one character; but if it doesn't, we'd miss the terminator character and include too much in the value. In principle this could lead to reading off the end of memory. It does not seem worth treating as a security issue though, because it would happen on client side, and besides client logic that's taking conninfo strings from untrusted sources has much worse security problems than this. Report and patch received off-list from Thomas Fanghaenel. Back-patch to 9.2 where the faulty code was introduced. http://git.postgresql.org/pg/commitdiff/b26e2081423cf1c70f83a11787351017c97cfd7c
  • Minor code beautification in conninfo_uri_parse_params(). Reading this made me itch, so clean the logic a bit. http://git.postgresql.org/pg/commitdiff/3d9b6f31eec150b5a6000e0814e81e36d9eb069a
  • Use FLEXIBLE_ARRAY_MEMBER for HeapTupleHeaderData.t_bits[]. This requires changing quite a few places that were depending on sizeof(HeapTupleHeaderData), but it seems for the best. Michael Paquier, some adjustments by me http://git.postgresql.org/pg/commitdiff/e1a11d93111ff3fba7a91f3f2ac0b0aca16909a8
  • Use FLEXIBLE_ARRAY_MEMBER in a number of other places. I think we're about done with this... http://git.postgresql.org/pg/commitdiff/2e211211a76782b6084194a5ced94c0795460047
  • Don't need to explain [1] kluge anymore in xfunc.sgml. http://git.postgresql.org/pg/commitdiff/0627eff3602c0ce0e8358d48ddb3ffa73963e4d0
  • Use FLEXIBLE_ARRAY_MEMBER in Windows-specific code. Be a tad more paranoid about overlength input, too. http://git.postgresql.org/pg/commitdiff/332f02f88beead6365bc2126c95451520bbfe163
  • Try to fix busted gettimeofday() code. Per buildfarm, we have to match the _stdcall property of the system functions. http://git.postgresql.org/pg/commitdiff/e9fd5545de3bb4efe163af4a9c957badac86ccd7
  • Represent BETWEEN as a special node type in raw parse trees. Previously, gram.y itself converted BETWEEN into AND (or AND/OR) nests of expression comparisons. This was always as bogus as could be, but fixing it hasn't risen to the top of the to-do list. The present patch invents an A_Expr representation for BETWEEN expressions, and does the expansion to comparison trees in parse_expr.c which is at least a slightly saner place to be doing semantic conversions. There should be no change in the post- parse-analysis results. This does nothing for the semantic issues with BETWEEN (dubious connection to btree-opclass semantics, and multiple evaluation of possibly volatile subexpressions) ... but it's a necessary preliminary step before we could fix any of that. The main immediate benefit is that preserving BETWEEN as an identifiable raw-parse-tree construct will enable better error messages. While at it, fix the code so that multiply-referenced subexpressions are physically duplicated before being passed through transformExpr(). This gets rid of one of the principal reasons why transformExpr() has historically had to allow already-processed input. http://git.postgresql.org/pg/commitdiff/34af082f95aa6adb8af5fbd4da46bd4c3c176856
  • Get rid of multiple applications of transformExpr() to the same tree. transformExpr() has for many years had provisions to do nothing when applied to an already-transformed expression tree. However, this was always ugly and of dubious reliability, so we'd be much better off without it. The primary historical reason for it was that gram.y sometimes returned multiple links to the same subexpression, which is no longer true as of my BETWEEN fixes. We'd also grown some lazy hacks in CREATE TABLE LIKE (failing to distinguish between raw and already-transformed index specifications) and one or two other places. This patch removes the need for and support for re-transforming already transformed expressions. The index case is dealt with by adding a flag to struct IndexStmt to indicate that it's already been transformed; which has some benefit anyway in that tablecmds.c can now Assert that transformation has happened rather than just assuming. The other main reason was some rather sloppy code for array type coercion, which can be fixed (and its performance improved too) by refactoring. I did leave transformJoinUsingClause() still constructing expressions containing untransformed operator nodes being applied to Vars, so that transformExpr() still has to allow Var inputs. But that's a much narrower, and safer, special case than before, since Vars will never appear in a raw parse tree, and they don't have any substructure to worry about. In passing fix some oversights in the patch that added CREATE INDEX IF NOT EXISTS (missing processing of IndexStmt.if_not_exists). These appear relatively harmless, but still sloppy coding practice. http://git.postgresql.org/pg/commitdiff/6a75562ed16b5fa52cfd8830e4546972e647db26
  • Add parse location fields to NullTest and BooleanTest structs. We did not need a location tag on NullTest or BooleanTest before, because no error messages referred directly to their locations. That's planned to change though, so add these fields in a separate housekeeping commit. Catversion bump because stored rules may change. http://git.postgresql.org/pg/commitdiff/c063da1769229efa8d8d21f0d068b3199ea3a6b3

Fujii Masao a poussé :

Heikki Linnakangas a poussé :

  • In the SSL test suite, use a root CA cert that won't expire (so quickly). All the other certificates were created to be valid for 10000 days, because we don't want to have to recreate them. But I missed the root CA cert, and the pre-created certificates included in the repository expired in January. Fix, and re-create all the certificates. http://git.postgresql.org/pg/commitdiff/2c75531a6cc49a56afbd5619c36b3daccbe243fa
  • Restore the SSL_set_session_id_context() call to OpenSSL renegotiation. This reverts the removal of the call in commit (272923a0). It turns out it wasn't superfluous after all: without it, renegotiation fails if a client certificate was used. The rest of the changes in that commit are still OK and not reverted. Per investigation of bug #12769 by Arne Scheffer, although this doesn't fix the reported bug yet. http://git.postgresql.org/pg/commitdiff/1c2b7c0879d83ff79e4adf2c0a883df92b713da4
  • Fix comment in libpq OpenSSL code about why a substitue BIO is used. The comment was copy-pasted from the backend code along with the implementation, but libpq has different reasons for using the BIO. http://git.postgresql.org/pg/commitdiff/c478959a009a926ec441f355219eae18537e648c
  • Fix knn-GiST queue comparison function to return heap tuples first. The part of the comparison function that was supposed to keep heap tuples ahead of index items was backwards. It would not lead to incorrect results, but it is more efficient to return heap tuples first, before scanning more index pages, when both have the same distance. Alexander Korotkov http://git.postgresql.org/pg/commitdiff/d17b6df239e6eebf288969e931cdbe8076d1fe12
  • Fix a bug in pairing heap removal code. After removal, the next_sibling pointer of a node was sometimes incorrectly left to point to another node in the heap, which meant that a node was sometimes linked twice into the heap. Surprisingly that didn't cause any crashes in my testing, but it was clearly wrong and could easily segfault in other scenarios. Also always keep the prev_or_parent pointer as NULL on the root node. That was not a correctness issue AFAICS, but let's be tidy. Add a debugging function, to dump the contents of a pairing heap as a string. It's #ifdef'd out, as it's not used for anything in any normal code, but it was highly useful in debugging this. Let's keep it handy for further reference. http://git.postgresql.org/pg/commitdiff/931bf3eb9b203ca02d729f5122a44cc250c27695
  • Remove dead structs. These are not used with the new WAL format anymore. GIN split records are simply always recorded as full-page images. Michael Paquier http://git.postgresql.org/pg/commitdiff/634618ecd039cf99fb8212b0314ef8a3754ca9c2

Kevin Grittner a poussé :

Robert Haas a poussé :

  • Improve pg_check_dir code and comments. Avoid losing errno if readdir() fails and closedir() works. Consistently return 4 rather than 3 if both a lost+found directory and other files are found, rather than returning one value or the other depending on the order of the directory listing. Update comments to match the actual behavior. These oversights date to commits 6f03927fce038096f53ca67eeab9adb24938f8a6 and 17f15239325a88581bb4f9cf91d38005f1f52d69. Marco Nenciarini http://git.postgresql.org/pg/commitdiff/5d6c2405f4bae6c87533d981d6a47587db501220
  • Don't require users of src/port/gettimeofday.c to initialize it. Commit 8001fe67a3d66c95861ce1f7075ef03953670d13 introduced this requirement, but per discussion, we want to avoid requirements of this type to make things easier on the calling code. An especially important consideration is that this may be used in frontend code, not just the backend. Asif Naeem, reviewed by Michael Paquier http://git.postgresql.org/pg/commitdiff/64235fecc6d377690444ca3702257b5e39ed3413

Álvaro Herrera a poussé :

  • Fix object identities for pg_conversion objects. We were neglecting to schema-qualify them. Backpatch to 9.3, where object identities were introduced as a concept by commit f8348ea32ec8. http://git.postgresql.org/pg/commitdiff/0d906798f680b2dd86cbc8d31ba64685aeb12092
  • Fix opclass/opfamily identity strings. The original representation uses "opcname for amname", which is good enough; but if we replace "for" with "using", we can apply the returned identity directly in a DROP command, as in DROP OPERATOR CLASS opcname USING amname This slightly simplifies code using object identities to programatically execute commands on these kinds of objects. Note backwards-incompatible change: The previous representation dates back to 9.3 when object identities were introduced by commit f8348ea3, but we don't want to change the behavior on released branches unnecessarily and so this is not backpatched. http://git.postgresql.org/pg/commitdiff/9c7dd350199fa030ccbd5538e1b8e13a9603fda4
  • Update PGSTAT_FILE_FORMAT_ID. Previous commit should have bumped it but didn't. Oops. Per note from Tom Lane. http://git.postgresql.org/pg/commitdiff/3b14bb77716a01915da4833194e7c8c1c51da66d
  • Have TRUNCATE update pgstat tuple counters. This works by keeping a per-subtransaction record of the ins/upd/del counters before the truncate, and then resetting them; this record is useful to return to the previous state in case the truncate is rolled back, either in a subtransaction or whole transaction. The state is propagated upwards as subtransactions commit. When the per-table data is sent to the stats collector, a flag indicates to reset the live/dead counters to zero as well. Catalog version bumped due to the change in pgstat format. Author: Alexander Shulgin Discussion: 1007.1207238291@sss.pgh.pa.us Discussion: 548F7D38.2000401@BlueTreble.com Reviewed-by: Álvaro Herrera, Jim Nasby http://git.postgresql.org/pg/commitdiff/d42358efb16cc81122c53ffb35ac381b9158e519
  • Remove unnecessary and unreliable test http://git.postgresql.org/pg/commitdiff/8902f79264d95ed84e4c9fb4749b3956b4c74349

Peter Eisentraut a poussé :

Andres Freund a poussé :

  • Force some system catalog table columns to be marked NOT NULL. In a manual pass over the catalog declaration I found a number of columns which the boostrap automatism didn't mark NOT NULL even though they actually were. Add BKI_FORCE_NOT_NULL markings to them. It's usually not critical if a system table column is falsely determined to be nullable as the code should always catch relevant cases. But it's good to have a extra layer in place. Discussion: 20150215170014.GE15326@awork2.anarazel.de http://git.postgresql.org/pg/commitdiff/82a532b34d7547b43b90e2e4d4953f4be1c655b8
  • Allow forcing nullness of columns during bootstrap. Bootstrap determines whether a column is null based on simple builtin rules. Those work surprisingly well, but nonetheless a few existing columns aren't set correctly. Additionally there is at least one patch sent to hackers where forcing the nullness of a column would be helpful. The boostrap format has gained FORCE [NOT] NULL for this, which will be emitted by genbki.pl when BKI_FORCE_(NOT_)?NULL is specified for a column in a catalog header. This patch doesn't change the marking of any existing columns. Discussion: 20150215170014.GE15326@awork2.anarazel.de http://git.postgresql.org/pg/commitdiff/eb68379c38202180bc8e33fb9987284e314b7fc8

Jeff Davis a poussé :

  • In array_agg(), don't create a new context for every group. Previously, each new array created a new memory context that started out at 8kB. This is incredibly wasteful when there are lots of small groups of just a few elements each. Change initArrayResult() and friends to accept a "subcontext" argument to indicate whether the caller wants the ArrayBuildState allocated in a new subcontext or not. If not, it can no longer be released separately from the rest of the memory context. Fixes bug report by Frank van Vugt on 2013-10-19. Tomas Vondra. Reviewed by Ali Akbar, Tom Lane, and me. http://git.postgresql.org/pg/commitdiff/b419865a814abbca12bdd6eef6a3d5ed67f432e1
  • Rename variable in AllocSetContextCreate to be consistent. Everywhere else in the file, "context" is of type MemoryContext and "set" is of type AllocSet. AllocSetContextCreate uses a variable of type AllocSet, so rename it from "context" to "set". http://git.postgresql.org/pg/commitdiff/74811c4050921959d54d42e2c15bb79f0e2c37f3

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Shigeru HANADA sent in two more revisions of a patch to implement join push-down support for foreign tables.
  • Michael Paquier sent in two more revisions of a patch to ensure that pg_dump gets attributes from tables in extensions.
  • Syed Rahila sent in another revision of a patch to implement compression of full-page writes.
  • Tom Lane sent in two more revisions of a patch to manipulate complex types as non-contiguous structures in-memory.
  • Peter Geoghegan sent in two more revisions of a patch to add INSERT ... ON CONFLICT {UPDATE | IGNORE}.
  • Amit Kapila sent in another revision of a patch to implement parallel sequential scans.
  • Petr (PJMODOS) Jelinek sent in two more revisions of a patch to implement TABLESAMPLE.
  • Ashutosh Bapat sent in a patch to implement some infrastructure for allowing 2PC across FDWs.
  • Michael Paquier sent in another revision of a patch to fix pg_dump's treatment of tables in EXTENSIONs.
  • Oskari Saarenmaa sent in another revision of a patch to fix aligned attribute for Sun CC and use custom macros for all attributes.
  • Alexander Korotkov sent in another revision of a patch to implement KNN-GiST with recheck.
  • Michael Paquier sent in another revision of a patch to enable table-level log_autovacuum_min_duration.
  • Stephen Frost sent in two more revisions of a patch to fix some odd behavior of updatable security barrier views on foreign tables.
  • Michael Paquier sent in two more revisions of a patch to fix a strange assertion using VACOPT_FREEZE in vacuum.c.
  • Kyotaro HORIGUCHI sent in three more revisions of a patch to fix a bug where pg_basebackup may fail to send feedback.
  • Robert Haas sent in another revision of a patch to allow assessing parallel safety.
  • Kyotaro HORIGUCHI sent in another revision of a patch to add regrole and regnamespace.
  • Adam Brightwell sent in another revision of a patch to remove the catupdate check.
  • Rugal Bernstein sent in a patch to fix psql/po/zh_CN.po for better comprehension by people in mainland China.
  • Corey Huinker sent in a patch to add polymorphic functions to dblink.
  • Etsuro Fujita sent in another revision of a patch to allow foreign tables to participate in table inheritance hierarchies.
  • Bruce Momjian sent in a patch to better document pg_upgrade on replicas.
  • David Fetter, Pavel Stehule, and Álvaro Herrera traded patches to fix a broken behavior in psql's \c for URIs and connect strings where only part of the specified item got used.
  • Álvaro Herrera sent in a patch to have RENAME routines return ObjectAddress rather than OID.
  • Álvaro Herrera sent in a patch to support more commands in event triggers.
  • Fujii Masao sent in another revision of a patch to add recovery_timeout option to control timeout of restore_command nonzero status code.
  • Andres Freund sent in a patch to debloat and deduplicate transaction commit/abort records.
  • Peter Geoghegan sent in a patch to make tuplesort_begin_datum() consistent with other comparable routines for other tuple cases.
  • Andres Freund sent in a WIP patch to resolve regtype/regproc in genbki.pl.
  • Andrew Dunstan sent in another revision of a patch to add min and max execute statement time in pg_stat_statement.
  • Andres Freund sent in a flock of patches to improve deparsing support.
  • Tom Lane sent in a patch to fix the precedence of standard comparison operators.
  • Peter Geoghegan sent in a patch to add a decay to the threshold that (estimated) abbreviated key cardinality must cross as a proportion of the (estimated) cardinality of the original set of strings to be sorted, while also quadrupling the initial required proportional cardinality to 20% of full key cardinality (but for just the first 10,000 strings, before this threshold is decayed aggressively).
  • Pavel Stehule sent in another revision of a patch to add array_position().

par N Bougain le mardi 24 février 2015 à 01h15

lundi 23 février 2015

Damien Clochard

PostgreSQL Dashboard

Un écran de suivi temps-réel basé sur Dashing et Sinatra

PostgreSQL Dashboard est outil de supervision très simple qui donne une vue instantanée de la santé d’une instance PostgreSQL.

L’outil est conçu pour être affiché sur un grand écran dans un salle de contrôle ou dans un espace de travail partagé.

Le tableau de bord actuel est composé de 5 “widgets” :

  • General Info : Version, nombre de bases, etc.
  • Hit Ratio : Le % de données trouvées en cache
  • Buffers : Le nombre de nouveaux buffers alloués
  • Queries : Le nombre de requête en cours sur l’instance
  • Twitter : Un aperçu du hashtag #PostgreSQL

Extensibilité

Ajouter un widget supplémentaire est très simple : cet outil est conçu pour faciliter l’écriture de widget “maison” et afficher des indicateurs métier.

Par ailleurs, la mise en page est totalement flexible. Vous pouvez simplement faire un drag’n’drop sur chaque widget et le placer à l’endroit qui vous convient. Le code HTML du tableau de bord peut être modifié pour des besoins spécifiques comme par exemple les dimensions d’un écran.

Liens & Remerciements

Certaines parties de PostgreSQL Dashboard sont basées sur le travail effectués pour d’autres outil PostgreSQL, notamment pgstats et pgcluu

PostgreSQL Dashboard est un projet ouvert distribué sous licence PostgreSQL. Toute contribution constructive est la bienvenue ! Vous pouvez envoyer vos idées, vos demandes et vos patches via les outils de GitHub.

Pour aller plus loin :

lundi 23 février 2015 à 21h27

dimanche 22 février 2015

Guillaume Lelarge

Durée d'établissement d'une connexion

Ce billet fait partie d'une série sur l'écriture de mon livre, « PostgreSQL - Architecture et notions avancées ».

J'ai toujours eu en tête qu'une connexion mettait du temps à s'établir entre un client et PostgreSQL. J'avais en tête un nombre qui me semblait plausible mais j'avoue que je n'avais jamais fait réellement le test.

Ce week-end, travaillant sur le chapitre sur la gestion des connexions, je me suis demandé si on pouvait calculer ce temps. J'ai donc regardé le code des processus postmaster/postgres pour ajouter quelques traces, histoire d'en savoir plus. Voici le patch que j'ai réalisé :

diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index f05114d..9d8fb8a 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2198,6 +2198,8 @@ ConnCreate(int serverFd)
 {
        Port       *port;
 
+       elog(LOG, "patch - ConnCreate(%d)", serverFd);
+
        if (!(port = (Port *) calloc(1, sizeof(Port))))
        {
                ereport(LOG,
@@ -3760,6 +3762,8 @@ BackendStartup(Port *port)
        Backend    *bn;                         /* for backend cleanup */
        pid_t           pid;
 
+       elog(LOG, "patch - BackendStart()");
+
        /*
         * Create backend data structure.  Better before the fork() so we can
         * handle failure cleanly.
@@ -3814,6 +3818,8 @@ BackendStartup(Port *port)
 
                MyProcPid = getpid();   /* reset MyProcPid */
 
+               elog(LOG, "patch - new pid is %d", MyProcPid);
+
                MyStartTime = time(NULL);
 
                /* We don't want the postmaster's proc_exit() handlers */
@@ -3916,6 +3922,8 @@ BackendInitialize(Port *port)
        char            remote_port[NI_MAXSERV];
        char            remote_ps_data[NI_MAXHOST];
 
+       elog(LOG, "patch - BackendInitialize()");
+
        /* Save port etc. for ps status */
        MyProcPort = port;
 
@@ -4096,6 +4104,8 @@ BackendRun(Port *port)
        int                     usecs;
        int                     i;
 
+       elog(LOG, "patch - BackendRun()");
+
        /*
         * Don't want backend to be able to see the postmaster random number
         * generator state.  We have to clobber the static random_seed *and* start
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index bc4eb33..4e1a3f7 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3578,6 +3578,7 @@ PostgresMain(int argc, char *argv[],
        sigjmp_buf      local_sigjmp_buf;
        volatile bool send_ready_for_query = true;
 
+       elog(LOG, "patch - PostgresMain()");
        /*
         * Initialize globals (already done if under postmaster, but not if
         * standalone).
@@ -3845,6 +3846,8 @@ PostgresMain(int argc, char *argv[],
         * were inside a transaction.
         */
 
+       elog(LOG, "patch - PostgresMain() - ready to execute command");
+
        if (sigsetjmp(local_sigjmp_buf, 1) != 0)
        {
                /*
@@ -4056,12 +4059,16 @@ PostgresMain(int argc, char *argv[],
                if (ignore_till_sync && firstchar != EOF)
                        continue;
 
+               elog(LOG, "patch - PostgresMain() - processing command");
+
                switch (firstchar)
                {
                        case 'Q':                       /* simple query */
                                {
                                        const char *query_string;
 
+                                       elog(LOG, "patch - PostgresMain() - executing simple query");
+
                                        /* Set statement_timestamp() */
                                        SetCurrentStatementStartTimestamp();
 
@@ -4279,6 +4286,8 @@ PostgresMain(int argc, char *argv[],
                        case 'X':
                        case EOF:
 
+                               elog(LOG, "patch - PostgresMain() - exiting");
+
                                /*
                                 * Reset whereToSendOutput to prevent ereport from attempting
                                 * to send any more messages to client.

En configurant PostgreSQL pour qu'il ajoute la date (à la milliseconde près) et le PID, et en configurant la trace des connexions et déconnexions :

log_min_duration_statement = 0
log_connections = on
log_disconnections = on
log_line_prefix = '%m [%p] '

et en exécutant la commande suivante :

$ psql -c "select * from t1 limit 200" b1

nous obtenons les traces suivantes :

2015-02-22 22:47:23.022 CET [6087] LOG:  patch - ConnCreate(5)
2015-02-22 22:47:23.022 CET [6087] LOG:  patch - BackendStart()
2015-02-22 22:47:23.023 CET [6283] LOG:  patch - new pid is 6283
2015-02-22 22:47:23.023 CET [6283] LOG:  patch - BackendInitialize()
2015-02-22 22:47:23.023 CET [6283] LOG:  connection received: host=[local]
2015-02-22 22:47:23.023 CET [6283] LOG:  patch - BackendRun()
2015-02-22 22:47:23.023 CET [6283] LOG:  patch - PostgresMain()
2015-02-22 22:47:23.025 CET [6283] LOG:  connection authorized: user=postgres database=b1
2015-02-22 22:47:23.027 CET [6283] LOG:  patch - PostgresMain() - ready to execute command
2015-02-22 22:47:23.027 CET [6283] LOG:  patch - PostgresMain() - processing command
2015-02-22 22:47:23.028 CET [6283] LOG:  patch - PostgresMain() - executing simple query
2015-02-22 22:47:23.028 CET [6283] LOG:  duration: 0.691 ms  statement: select * from t1 limit 200
2015-02-22 22:47:23.736 CET [6283] LOG:  patch - PostgresMain() - processing command
2015-02-22 22:47:23.736 CET [6283] LOG:  patch - PostgresMain() - exiting
2015-02-22 22:47:23.737 CET [6283] LOG:  disconnection: session time: 0:00:00.913 user=postgres database=b1 host=[local]

Autrement dit, il faut compter quelques millisecondes pour établir une connexion sans pooler. Après différents tests (impliquant notamment pgbench), le pire que j'ai vu est 10 millisecondes. Pas bien méchant quand on y pense. J'ai aussi noté que la toute première connexion était bien plus lente (dans les 40 millisecondes), ce qui reste encore bien loin de ce que j'imaginais.

J'ai aussi testé avec différentes valeurs du shared_buffers car il semblerait que la taille mémoire d'un processus a une importance dans la durée d'exécution de l'appel système fork().

Comme quoi il est vraiment préférable de tout tester pour ne pas avoir d'idées préconçues.

par Guillaume Lelarge le dimanche 22 février 2015 à 21h49

jeudi 19 février 2015

Guillaume Lelarge

PostgreSQL et la mémoire partagée

Ce billet fait partie d'une série sur l'écriture de mon livre, « PostgreSQL - Architecture et notions avancées ».

Lors de l'écriture du chapitre sur la gestion de la mémoire par PostgreSQL, j'ai cherché à différencier les différents blocs alloués en mémoire partagée. La documentation de PostgreSQL est assez peu bavard sur ce sujet, je me suis donc retourné vers le code source. Ce dernier donne énormément d'informations à qui se donne un peu la peine de les chercher. J'ai fini par trouver le fichier src/backend/storage/ipc/shmem.c qui s'occupe de la gestion de la mémoire partagée (shmem pour SHared MEMory).

Ce fichier contient différentes fonctions, dont la fonction ShmemInitStruct, dont le but est d'initialiser une structure en mémoire partagée. Il suffit de lui fournir le nom de la structure et sa taille, et la fonction se charge de l'allocation. Une petite modification de cette fonction permet d'afficher quelques informations à chaque appel de cette fonction. Voici ce patch :

diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 2ea2216..8703f48 100644
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -402,6 +402,8 @@ ShmemInitStruct(const char *name, Size size, bool *foundPtr)
 	{
 		/* It isn't in the table yet. allocate and initialize it */
 		structPtr = ShmemAlloc(size);
+		ereport(LOG, (errmsg("allocate shared memory segment "%s", size %d",
+				name, (int)size)));
 		if (structPtr == NULL)
 		{
 			/* out of memory; remove the failed ShmemIndex entry */

Au lancement de PostgreSQL, modifié avec ce patch, nous obtenons cette sortie dans les traces :

LOG:  allocate shared memory segment "Control File", size 248 
LOG:  allocate shared memory segment "XLOG Ctl", size 16804496
LOG:  allocate shared memory segment "CLOG Ctl", size 525312
LOG:  allocate shared memory segment "SUBTRANS Ctl", size 263168
LOG:  allocate shared memory segment "MultiXactOffset Ctl", size 65856
LOG:  allocate shared memory segment "MultiXactMember Ctl", size 131648
LOG:  allocate shared memory segment "Shared MultiXact State", size 176 
LOG:  allocate shared memory segment "Buffer Descriptors", size 33554432
LOG:  allocate shared memory segment "Buffer Blocks", size 0
LOG:  allocate shared memory segment "Shared Buffer Lookup Table", size 32880
LOG:  allocate shared memory segment "Buffer Strategy Status", size 32
LOG:  allocate shared memory segment "LOCK hash", size 2160
LOG:  allocate shared memory segment "PROCLOCK hash", size 2160
LOG:  allocate shared memory segment "Fast Path Strong Relation Lock Data", size 4100
LOG:  allocate shared memory segment "PREDICATELOCKTARGET hash", size 2160
LOG:  allocate shared memory segment "PREDICATELOCK hash", size 2160
LOG:  allocate shared memory segment "PredXactList", size 88
LOG:  allocate shared memory segment "SERIALIZABLEXID hash", size 2160
LOG:  allocate shared memory segment "RWConflictPool", size 24
LOG:  allocate shared memory segment "FinishedSerializableTransactions", size 16
LOG:  allocate shared memory segment "OldSerXid SLRU Ctl", size 131648
LOG:  allocate shared memory segment "OldSerXidControlData", size 16
LOG:  allocate shared memory segment "Proc Header", size 88
LOG:  allocate shared memory segment "Proc Array", size 108 
LOG:  allocate shared memory segment "Backend Status Array", size 3672
LOG:  allocate shared memory segment "Backend Application Name Buffer", size 1088
LOG:  allocate shared memory segment "Backend Client Host Name Buffer", size 1088
LOG:  allocate shared memory segment "Backend Activity Buffer", size 17408
LOG:  allocate shared memory segment "Prepared Transaction Table", size 16
LOG:  allocate shared memory segment "Background Worker Data", size 1992
LOG:  allocate shared memory segment "shmInvalBuffer", size 66104
LOG:  allocate shared memory segment "PMSignalState", size 180 
LOG:  allocate shared memory segment "ProcSignalSlots", size 864 
LOG:  allocate shared memory segment "Checkpointer Data", size 10485800
LOG:  allocate shared memory segment "AutoVacuum Data", size 224 
LOG:  allocate shared memory segment "Wal Sender Ctl", size 56
LOG:  allocate shared memory segment "Wal Receiver Ctl", size 1192
LOG:  allocate shared memory segment "BTree Vacuum State", size 216 
LOG:  allocate shared memory segment "Sync Scan Locations List", size 656 
LOG:  allocate shared memory segment "Async Queue Control", size 244 
LOG:  allocate shared memory segment "Async Ctl", size 65856
LOG:  allocate shared memory segment "pg_stat_statements", size 48
LOG:  allocate shared memory segment "pg_stat_statements hash", size 2160

(Et comme on peut le constater par les deux dernières lignes, ce serveur avait l'extension pg_stat_statements à charger)

Nous connaissons ainsi les différents segments et leur taille respective. À partir de là, il a suffit de suivre la piste des différents segments pour comprendre leur utilité et la façon dont ils sont utilisés.

par Guillaume Lelarge le jeudi 19 février 2015 à 22h53

mercredi 18 février 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 15 février 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en février

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Fujii Masao a poussé :

  • Move pg_lzcompress.c to src/common. The meta data of PGLZ symbolized by PGLZ_Header is removed, to make the compression and decompression code independent on the backend-only varlena facility. PGLZ_Header is being used to store some meta data related to the data being compressed like the raw length of the uncompressed record or some varlena-related data, making it unpluggable once PGLZ is stored in src/common as it contains some backend-only code paths with the management of varlena structures. The APIs of PGLZ are reworked at the same time to do only compression and decompression of buffers without the meta-data layer, simplifying its use for a more general usage. On-disk format is preserved as well, so there is no incompatibility with previous major versions of PostgreSQL for TOAST entries. Exposing compression and decompression APIs of pglz makes possible its use by extensions and contrib modules. Especially this commit is required for upcoming WAL compression feature so that the WAL reader facility can decompress the WAL data by using pglz_decompress. Michael Paquier, reviewed by me. http://git.postgresql.org/pg/commitdiff/40bede5477bb5bce98ce9548841cb414634c26f7

Heikki Linnakangas a poussé :

  • Fix MSVC build. When I moved pg_crc.c from src/port to src/common, I forgot to modify MSVC build script accordingly. http://git.postgresql.org/pg/commitdiff/cc761b170c5e7b4ef22ed918f4785ec1fabe62cd
  • Move pg_crc.c to src/common, and remove pg_crc_tables.h. To get CRC functionality in a client program, you now need to link with libpgcommon instead of libpgport. The CRC code has nothing to do with portability, so libpgcommon is a better home. (libpgcommon didn't exist when pg_crc.c was originally moved to src/port.) Remove the possibility to get CRC functionality by just #including pg_crc_tables.h. I'm not aware of any extensions that actually did that and couldn't simply link with libpgcommon. This also moves the pg_crc.h header file from src/include/utils to src/include/common, which will require changes to any external programs that currently does #include "utils/pg_crc.h". That seems acceptable, as include/common is clearly the right home for it now, and the change needed to any such programs is trivial. http://git.postgresql.org/pg/commitdiff/c619c2351f7ec429b6ddce519c939f7b8465d711
  • Speed up CRC calculation using slicing-by-8 algorithm. This speeds up WAL generation and replay. The new algorithm is significantly faster with large inputs, like full-page images or when inserting wide rows. It is slower with tiny inputs, i.e. less than 10 bytes or so, but the speedup with longer inputs more than make up for that. Even small WAL records at least have 24 byte header in the front. The output is identical to the current byte-at-a-time computation, so this does not affect compatibility. The new algorithm is only used for the CRC-32C variant, not the legacy version used in tsquery or the "traditional" CRC-32 used in hstore and ltree. Those are not as performance critical, and are usually only applied over small inputs, so it seems better to not carry around the extra lookup tables to speed up those rare cases. Abhijit Menon-Sen http://git.postgresql.org/pg/commitdiff/025c02420de990c15a90e9e3f86fcfbc5b59ee88
  • Simplify waiting logic in reading from / writing to client. The client socket is always in non-blocking mode, and if we actually want blocking behaviour, we emulate it by sleeping and retrying. But we have retry loops at different layers for reads and writes, which was confusing. To simplify, remove all the sleeping and retrying code from the lower levels, from be_tls_read and secure_raw_read and secure_raw_write, and put all the logic in secure_read() and secure_write(). http://git.postgresql.org/pg/commitdiff/80788a431e9bff06314a054109fdea66ac538199
  • Simplify the way OpenSSL renegotiation is initiated in server. At least in all modern versions of OpenSSL, it is enough to call SSL_renegotiate() once, and then forget about it. Subsequent SSL_write() and SSL_read() calls will finish the handshake. The SSL_set_session_id_context() call is unnecessary too. We only have one SSL context, and the SSL session was created with that to begin with. http://git.postgresql.org/pg/commitdiff/272923a0a6956187471df4f032eee06559520390
  • Fix broken #ifdef for __sparcv8. Rob Rowan. Backpatch to all supported versions, like the patch that added the broken #ifdef. http://git.postgresql.org/pg/commitdiff/33e879c4e9f485405fc19b663392d8635714d020

Tom Lane a poussé :

  • Minor cleanup/code review for "indirect toast" stuff. Fix some issues I noticed while fooling with an extension to allow an additional kind of toast pointer. Much of this is just comment improvement, but there are a couple of actual bugs, which might or might not be reachable today depending on what can happen during logical decoding. An example is that toast_flatten_tuple() failed to cover the possibility of an indirection pointer in its input. Back-patch to 9.4 just in case that is reachable now. In HEAD, also correct some really minor issues with recent compression reorganization, such as dangerously underparenthesized macros. http://git.postgresql.org/pg/commitdiff/bc4de01db3a210e988dc88b585d7c38e6e7054c7
  • Fix GEQO to not assume its join order heuristic always works. Back in commit 400e2c934457bef4bc3cc9a3e49b6289bd761bc0 I rewrote GEQO's gimme_tree function to improve its heuristic for modifying the given tour into a legal join order. In what can only be called a fit of hubris, I supposed that this new heuristic would *always* find a legal join order, and ripped out the old logic that allowed gimme_tree to sometimes fail. The folly of this is exposed by bug #12760, in which the "greedy" clumping behavior of merge_clump() can lead it into a dead end which could only be recovered from by un-clumping. We have no code for that and wouldn't know exactly what to do with it if we did. Rather than try to improve the heuristic rules still further, let's just recognize that it *is* a heuristic and probably must always have failure cases. So, put back the code removed in the previous commit to allow for failure (but comment it a bit better this time). It's possible that this code was actually fully correct at the time and has only been broken by the introduction of LATERAL. But having seen this example I no longer have much faith in that proposition, so back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/1a179f36f7c612d3560a72396c559cd876dfab6b
  • Fix pg_dump's heuristic for deciding which casts to dump. Back in 2003 we had a discussion about how to decide which casts to dump. At the time pg_dump really only considered an object's containing schema to decide what to dump (ie, dump whatever's not in pg_catalog), and so we chose a complicated idea involving whether the underlying types were to be dumped (cf commit a6790ce85752b67ad994f55fdf1a450262ccc32e). But users are allowed to create casts between built-in types, and we failed to dump such casts. Let's get rid of that heuristic, which has accreted even more ugliness since then, in favor of just looking at the cast's OID to decide if it's a built-in cast or not. In passing, also fix some really ancient code that supposed that it had to manufacture a dependency for the cast on its cast function; that's only true when dumping from a pre-7.3 server. This just resulted in some wasted cycles and duplicate dependency-list entries with newer servers, but we might as well improve it. Per gripes from a number of people, most recently Greg Sabino Mullane. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/9feefedf9e92066fa6609d1e1e17b4892d81716f
  • Fix more memory leaks in failure path in buildACLCommands. We already had one go at this issue in commit d73b7f973db5ec7e, but we failed to notice that buildACLCommands also leaked several PQExpBuffers along with a simply malloc'd string. This time let's try to make the fix a bit more future-proof by eliminating the separate exit path. It's still not exactly critical because pg_dump will curl up and die on failure; but since the amount of the potential leak is now several KB, it seems worth back-patching as far as 9.2 where the previous fix landed. Per Coverity, which evidently is smarter than clang's static analyzer. http://git.postgresql.org/pg/commitdiff/9179444d070713f53a97bc8f5c898a8304424c4d
  • Fix minor memory leak in ident_inet(). We'd leak the ident_serv data structure if the second pg_getaddrinfo_all (the one for the local address) failed. This is not of great consequence because a failure return here just leads directly to backend exit(), but if this function is going to try to clean up after itself at all, it should not have such holes in the logic. Try to fix it in a future-proof way by having all the failure exits go through the same cleanup path, rather than "optimizing" some of them. Per Coverity. Back-patch to 9.2, which is as far back as this patch applies cleanly. http://git.postgresql.org/pg/commitdiff/58146d35deed3bd3e05ce4b5ba529e4e4a369ddb
  • Fix missing PQclear() in libpqrcv_endstreaming(). This omission leaked one PGresult per WAL streaming cycle, which possibly would never be enough to notice in the real world, but it's still a leak. Per Coverity. Back-patch to 9.3 where the error was introduced. http://git.postgresql.org/pg/commitdiff/4f38a281a3d4a7cbcf9de2f676d44336d57b7717
  • Avoid returning undefined bytes in chkpass_in(). We can't really fix the problem that the result is defined to depend on random(), so it is still going to fail the "unstable input conversion" test in parse_type.c. However, we can at least satify valgrind. (It looks like this code used to be valgrind-clean, actually, until somebody did a careless s/strncpy/strlcpy/g on it.) In passing, let's just make real sure that chkpass_out doesn't overrun its output buffer. No need for backpatch, I think, since this is just to satisfy debugging tools. Asif Naeem http://git.postgresql.org/pg/commitdiff/80986e85aafd64165efc7f45b330f2f213713032

Michael Meskes a poussé :

Andres Freund a poussé :

Bruce Momjian a poussé :

Peter Eisentraut a poussé :

Andrew Dunstan a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Tatsuo Ishii and Michael Paquier traded patches to fix some incompatible options in pgbench.
  • David Steele sent in two more revisions of a patch to implement pgaudit.
  • Jan Urbański sent in two revisions of a patch to fix broken multi-threaded SSL callback handling in libpq.
  • Rahila Syed sent in three more revisions of a patch to implement compression of full-page writes.
  • Peter Geoghegan and Heikki Linnakangas traded patches to implement INSERT ... ON CONFLICT {UPDATE | IGNORE}.
  • Heikki Linnakangas sent in two revisions of a patch to use use Intel SSE4.2 CRC instructions where available.
  • Michael Paquier sent in three more revisions of a patch to implement table-level autovacuum_min_duration.
  • Michael Paquier sent in a patch to add a wal_retrieve_retry_interval.
  • Kyotaro HORIGUCHI sent in two revisions of a patch to fix an issue where pg_basebackup fails silently.
  • Tom Lane sent in two revisions of a patch to manipulate complex types as non-contiguous structures in-memory.
  • Etsuro Fujita sent in another revision of a patch to fix ExplainModifyTarget to work as expected.
  • Michael Paquier sent in a patch to add palloc_extended for frontend and backend, and another to rework handling of OOM when allocating record buffer in XLOG reader.
  • Anastasia Lubennikova sent in two more revisions of a patch to allow index-only scans for GiST indexes.
  • Emre Hasegeli sent in another revision of a patch to add a BRIN range operator class.
  • Robert Haas sent in two more revisions of a patch to implement parallel mode and parallel contexts.
  • Marco Nenciarini sent in another revision of a patch to implement file-based incremental backup.
  • Marco Nenciarini sent in another revision of a patch to fix pg_check_dir.
  • Michael Paquier sent in a patch to fix an odd assertion using VACOPT_FREEZE in vacuum.c.
  • Teodor Sigaev sent in a patch to implement some of the intarray features for all arrays.
  • Teodor Sigaev sent in another revision of a patch to add a compress method for SP-GiST.
  • Heikki Linnakangas sent in another revision of a patch to redesign checkpoint_segments.
  • Heikki Linnakangas sent in a patch to refactor unit conversions.
  • Atri Sharma sent in another revision of a patch to implement UPDATE ... SET(*) =.
  • Asif Naeem sent in a patch to fix chkpass to work under RANDOMIZE_ALLOCATED_MEMORY.
  • Kevin Grittner sent in two revisions of a patch to fix an issue manifesting as "multiple backends attempting to wait for pincount 1".
  • Kevin Grittner sent in a patch to help control bloat in the face of old snapshots.
  • Peter Eisentraut sent in a patch to speed up make check-world.
  • Álvaro Herrera sent in another revision of a patch to add CREATE ALTER, GRANT/REVOKE, COMMENT ON, and SECURITY LABEL command deparsing support to event triggers.
  • Peter Eisentraut sent in a patch to add a pg_settings.pending_restart column.
  • Peter Eisentraut sent in a patch to use forward slashes exclusively in Windows builds, making it possible at least in theory to build such on non-Windows machines.
  • Kevin Grittner sent in a patch to allow a "snapshot too old" error.
  • Andrew Dunstan sent in two revisions of a patch to add mogrify and indent to JSONB.
  • Robert Haas sent in another revision of a patch to allow assessing parallel safety.
  • Alexander Korotkov sent in another revision of a patch to implement KNN-GiST with recheck.
  • Kaigai Kouhei sent in another revision of a patch to implement join replacement with foreign-/custom-scan.
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to implement a sequence access method.
  • Tomas Vondra sent in a PoC patch to implement a mmap-based memory context / allocator.
  • Emre Hasegeli sent in another revision of a patch to implement selectivity estimation for inet operators.
  • Andres Freund sent in another revision of a patch to implement replication identifiers.
  • David Steele sent in a patch to fix some dependency issues in the documentation build for OSX.
  • Andrew (RhodiumToad) Gierth and Tomas Vondra traded patches to fix some really bad blowups with hash outer join and nulls.

par N Bougain le mercredi 18 février 2015 à 01h54

Nouvelles hebdomadaires de PostgreSQL - 8 février 2015

Correctifs de sécurité : 9.4.1, 9.3.6, 9.2.10, 9.1.15 et 9.0.19. Mettez à jour ! http://www.postgresql.org/about/news/1569/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en février

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Peter Eisentraut a poussé :

Tom Lane a poussé :

  • Doc: fix syntax description for psql's \setenv. The variable name isn't optional --- looks like a copy-and-paste-o from the \set command, where it is. Dilip Kumar http://git.postgresql.org/pg/commitdiff/f9ee8ea10a432bd5692ef9ff25055717fbf290ce
  • Fix breakage in GEODEBUG debug code. LINE doesn't have an "m" field (anymore anyway). Also fix unportable assumption that %x can print the result of pointer subtraction. In passing, improve single_decode() in minor ways: Remove unnecessary leading-whitespace skip (strtod does that already). Make GEODEBUG message more intelligible. Remove entirely-useless test to see if strtod returned a silly pointer. Don't bother computing trailing-whitespace skip unless caller wants an ending pointer. This has been broken since 261c7d4b653bc3e44c31fd456d94f292caa50d8f. Although it's only debug code, might as well fix the 9.4 branch too. http://git.postgresql.org/pg/commitdiff/778d498c7d9099c784f14199a319ec2a84d91191
  • Remove unused "m" field in LSEG. This field has been unreferenced since 1998, and does not appear in lseg values stored on disk (since sizeof(lseg) is only 32 bytes according to pg_type). There was apparently some idea of maintaining it just in values appearing in memory, but the bookkeeping required to make that work would surely far outweigh the cost of recalculating the line's slope when needed. Remove it to (a) simplify matters and (b) suppress some uninitialized-field whining from Coverity. http://git.postgresql.org/pg/commitdiff/cec916f35be5531efdaf721a46313feb36e0cd76

Noah Misch a poussé :

  • Prevent Valgrind Memcheck errors around px_acquire_system_randomness(). This function uses uninitialized stack and heap buffers as supplementary entropy sources. Mark them so Memcheck will not complain. Back-patch to 9.4, where Valgrind Memcheck cooperation first appeared. Marko Tiikkaja http://git.postgresql.org/pg/commitdiff/59b919822ab060f721e235964d19b55a19c815f0
  • Cherry-pick security-relevant fixes from upstream imath library. This covers alterations to buffer sizing and zeroing made between imath 1.3 and imath 1.20. Valgrind Memcheck identified the buffer overruns and reliance on uninitialized data; their exploit potential is unknown. Builds specifying --with-openssl are unaffected, because they use the OpenSSL BIGNUM facility instead of imath. Back-patch to 9.0 (all supported versions). Security: CVE-2015-0243 http://git.postgresql.org/pg/commitdiff/8b59672d8d23ea4203cf2701d126a96edca5bdd6
  • Fix buffer overrun after incomplete read in pullf_read_max(). Most callers pass a stack buffer. The ensuing stack smash can crash the server, and we have not ruled out the viability of attacks that lead to privilege escalation. Back-patch to 9.0 (all supported versions). Marko Tiikkaja Security: CVE-2015-0243 http://git.postgresql.org/pg/commitdiff/1dc75515868454c645ded22d38054ec693e23ec6
  • Check DCH_MAX_ITEM_SIZ limits with <=, not <. We reserve space for the full amount, not one less. The affected checks deal with localized month and day names. Today's DCH_MAX_ITEM_SIZ value would suffice for a 60-byte day name, while the longest known is the 49-byte mn_CN.utf-8 word for "Saturday." Thus, the upshot of this change is merely to avoid misdirecting future readers of the code; users are not expected to see errors either way. http://git.postgresql.org/pg/commitdiff/237795a7b4ccf359423bdd99476782d4880eda83
  • Assert(PqCommReadingMsg) in pq_peekbyte(). Interrupting pq_recvbuf() can break protocol sync, so its callers all deserve this assertion. The one pq_peekbyte() caller suffices already. http://git.postgresql.org/pg/commitdiff/a7a4adcf8d1e5a34a15ad86aee7d6aa0b8c15d43

Robert Haas a poussé :

Heikki Linnakangas a poussé :

  • Remove dead code. Commit 13629df changed metaphone() function to return an empty string on empty input, but it left the old error message in place. It's now dead code. Michael Paquier, per Coverity warning. http://git.postgresql.org/pg/commitdiff/4eaafa045362298adfe61fdd83b47b22211c75d1
  • Fix typo in comment. Amit Langote http://git.postgresql.org/pg/commitdiff/efba7a542fd52ed7607f1d8b162d898da0baa22c
  • Rephrase the documentation on pg_receivexlog --synchronous option. The old wording talked about a "sync command", meaining fsync(), but it was not very clear. http://git.postgresql.org/pg/commitdiff/507627f5b5c009aa95c04254614778c6263a3dc2
  • Refactor page compactifying code. The logic to compact away removed tuples from page was duplicated with small differences in PageRepairFragmentation, PageIndexMultiDelete, and PageIndexDeleteNoCompact. Put it into a common function. Reviewed by Peter Geoghegan. http://git.postgresql.org/pg/commitdiff/809d9a260b5d068a6a366273cd724bf9b396b026
  • Add API functions to libpq to interrogate SSL related stuff. This makes it possible to query for things like the SSL version and cipher used, without depending on OpenSSL functions or macros. That is a good thing if we ever get another SSL implementation. PQgetssl() still works, but it should be considered as deprecated as it only works with OpenSSL. In particular, PQgetSslInUse() should be used to check if a connection uses SSL, because as soon as we have another implementation, PQgetssl() will return NULL even if SSL is in use. http://git.postgresql.org/pg/commitdiff/91fa7b4719ac583420d9143132ba4ccddefbc5b2
  • Add dummy PQsslAttributes function for non-SSL builds. All the other new SSL information functions had dummy versions in be-secure.c, but I missed PQsslAttributes(). Oops. Surprisingly, the linker did not complain about the missing function on most platforms represented in the buildfarm, even though it is exported, except for a few Windows systems. http://git.postgresql.org/pg/commitdiff/302262d52111ce71dececfe18448792cee415a57
  • Fix memory leaks on OOM in ecpg. These are fairly obscure cases, but let's keep Coverity happy. Michael Paquier with some further fixes by me. http://git.postgresql.org/pg/commitdiff/c31b5d9ddf25d26dac5f5c215b8a5e8b3dfa37e6
  • Fix reference-after-free when waiting for another xact due to constraint. If an insertion or update had to wait for another transaction to finish, because there was another insertion with conflicting key in progress, we would pass a just-free'd item pointer to XactLockTableWait(). All calls to XactLockTableWait() and MultiXactIdWait() had similar issues. Some passed a pointer to a buffer in the buffer cache, after already releasing the lock. The call in EvalPlanQualFetch had already released the pin too. All but the call in execUtils.c would merely lead to reporting a bogus ctid, however (or an assertion failure, if enabled). All the callers that passed HeapTuple->t_data->t_ctid were slightly bogus anyway: if the tuple was updated (again) in the same transaction, its ctid field would point to the next tuple in the chain, not the tuple itself. Backpatch to 9.4, where the 'ctid' argument to XactLockTableWait was added (in commit f88d4cfc) http://git.postgresql.org/pg/commitdiff/57fe246890ad51e166fb6a8da937e41c35d7a279
  • Use a separate memory context for GIN scan keys. It was getting tedious to track and release all the different things that form a scan key. We were leaking at least the queryCategories array, and possibly more, on a rescan. That was visible if a GIN index was used in a nested loop join. This also protects from leaks in extractQuery method. No backpatching, given the lack of complaints from the field. Maybe later, after this has received more field testing. http://git.postgresql.org/pg/commitdiff/d88976cfa1302e8dccdcbfe55e9e29faee8c0cdf
  • Report WAL flush, not insert, position in replication IDENTIFY_SYSTEM. When beginning streaming replication, the client usually issues the IDENTIFY_SYSTEM command, which used to return the current WAL insert position. That's not suitable for the intended purpose of that field, however. pg_receivexlog uses it to start replication from the reported point, but if it hasn't been flushed to disk yet, it will fail. Change IDENTIFY_SYSTEM to report the flush position instead. Backpatch to 9.1 and above. 9.0 doesn't report any WAL position. http://git.postgresql.org/pg/commitdiff/ff16b40f8cac67a740f4edaa9fc2d9c7ba44c73a
  • Be more careful to not lose sync in the FE/BE protocol. If any error occurred while we were in the middle of reading a protocol message from the client, we could lose sync, and incorrectly try to interpret a part of another message as a new protocol message. That will usually lead to an "invalid frontend message" error that terminates the connection. However, this is a security issue because an attacker might be able to deliberately cause an error, inject a Query message in what's supposed to be just user data, and have the server execute it. We were quite careful to not have CHECK_FOR_INTERRUPTS() calls or other operations that could ereport(ERROR) in the middle of processing a message, but a query cancel interrupt or statement timeout could nevertheless cause it to happen. Also, the V2 fastpath and COPY handling were not so careful. It's very difficult to recover in the V2 COPY protocol, so we will just terminate the connection on error. In practice, that's what happened previously anyway, as we lost protocol sync. To fix, add a new variable in pqcomm.c, PqCommReadingMsg, that is set whenever we're in the middle of reading a message. When it's set, we cannot safely ERROR out and continue running, because we might've read only part of a message. PqCommReadingMsg acts somewhat similarly to critical sections in that if an error occurs while it's set, the error handler will force the connection to be terminated, as if the error was FATAL. It's not implemented by promoting ERROR to FATAL in elog.c, like ERROR is promoted to PANIC in critical sections, because we want to be able to use PG_TRY/CATCH to recover and regain protocol sync. pq_getmessage() takes advantage of that to prevent an OOM error from terminating the connection. To prevent unnecessary connection terminations, add a holdoff mechanism similar to HOLD/RESUME_INTERRUPTS() that can be used hold off query cancel interrupts, but still allow die interrupts. The rules on which interrupts are processed when are now a bit more complicated, so refactor ProcessInterrupts() and the calls to it in signal handlers so that the signal handlers always call it if ImmediateInterruptOK is set, and ProcessInterrupts() can decide to not do anything if the other conditions are not met. Reported by Emil Lenngren. Patch reviewed by Noah Misch and Andres Freund. Backpatch to all supported versions. Security: CVE-2015-0244 http://git.postgresql.org/pg/commitdiff/2b3a8b20c2da9f39ffecae25ab7c66974fbc0d3b

Andres Freund a poussé :

  • Use a nonblocking socket for FE/BE communication and block using latches. This allows to introduce more elaborate handling of interrupts while reading from a socket. Currently some interrupt handlers have to do significant work from inside signal handlers, and it's very hard to correctly write code to do so. Generic signal handler limitations, combined with the fact that we can't safely jump out of a signal handler while reading from the client have prohibited implementation of features like timeouts for idle-in-transaction. Additionally we use the latch code to wait in a couple places where we previously only had waiting code on windows as other platforms just busy looped. This can increase the number of systemcalls happening during FE/BE communication. Benchmarks so far indicate that the impact isn't very high, and there's room for optimization in the latch code. The chance of cleaning up the usage of latches gives us, seem to outweigh the risk of small performance regressions. This commit theoretically can't used without the next patch in the series, as WaitLatchOrSocket is not defined to be fully signal safe. As we already do that in some cases though, it seems better to keep the commits separate, so they're easier to understand. Author: Andres Freund Reviewed-By: Heikki Linnakangas http://git.postgresql.org/pg/commitdiff/387da18874afa17156ee3af63766f17efb53c4b9
  • Introduce and use infrastructure for interrupt processing during client reads. Up to now large swathes of backend code ran inside signal handlers while reading commands from the client, to allow for speedy reaction to asynchronous events. Most prominently shared invalidation and NOTIFY handling. That means that complex code like the starting/stopping of transactions is run in signal handlers... The required code was fragile and verbose, and is likely to contain bugs. That approach also severely limited what could be done while communicating with the client. As the read might be from within openssl it wasn't safely possible to trigger an error, e.g. to cancel a backend in idle-in-transaction state. We did that in some cases, namely fatal errors, nonetheless. Now that FE/BE communication in the backend employs non-blocking sockets and latches to block, we can quite simply interrupt reads from signal handlers by setting the latch. That allows us to signal an interrupted read, which is supposed to be retried after returning from within the ssl library. As signal handlers now only need to set the latch to guarantee timely interrupt processing, remove a fair amount of complicated & fragile code from async.c and sinval.c. We could now actually start to process some kinds of interrupts, like sinval ones, more often that before, but that seems better done separately. This work will hopefully allow to handle cases like being blocked by sending data, interrupting idle transactions and similar to be implemented without too much effort. In addition to allowing getting rid of ImmediateInterruptOK, that is. Author: Andres Freund Reviewed-By: Heikki Linnakangas http://git.postgresql.org/pg/commitdiff/4f85fde8eb860f263384fffdca660e16e77c7f76
  • Process 'die' interrupts while reading/writing from the client socket. Up to now it was impossible to terminate a backend that was trying to send/recv data to/from the client when the socket's buffer was already full/empty. While the send/recv calls itself might have gotten interrupted by signals on some platforms, we just immediately retried. That could lead to situations where a backend couldn't be terminated , after a client died without the connection being closed, because it was blocked in send/recv. The problem was far more likely to be hit when sending data than when reading. That's because while reading a command from the client, and during authentication, we processed interrupts immediately . That primarily left COPY FROM STDIN as being problematic for recv. Change things so that that we process 'die' events immediately when the appropriate signal arrives. We can't sensibly react to query cancels at that point, because we might loose sync with the client as we could be in the middle of writing a message. We don't interrupt writes if the write buffer isn't full, as indicated by write() returning EWOULDBLOCK, as that would lead to fewer error messages reaching clients. Per discussion with Kyotaro HORIGUCHI and Heikki Linnakangas Discussion: 20140927191243.GD5423@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/4fe384bd858671d40d311ca68cc9b80f4c683a3e
  • Don't allow immediate interrupts during authentication anymore. We used to handle authentication_timeout by setting ImmediateInterruptOK to true during large parts of the authentication phase of a new connection. While that happens to work acceptably in practice, it's not particularly nice and has ugly corner cases. Previous commits converted the FE/BE communication to use latches and implemented support for interrupt handling during both send/recv. Building on top of that work we can get rid of ImmediateInterruptOK during authentication, by immediately treating timeouts during authentication as a reason to die. As die interrupts are handled immediately during client communication that provides a sensibly quick reaction time to authentication timeout. Additionally add a few CHECK_FOR_INTERRUPTS() to some more complex authentication methods. More could be added, but this already should provides a reasonable coverage. While it this overall increases the maximum time till a timeout is reacted to, it greatly reduces complexity and increases reliability. That seems like a overall win. If the increase proves to be noticeable we can deal with those cases by moving to nonblocking network code and add interrupt checking there. Reviewed-By: Heikki Linnakangas http://git.postgresql.org/pg/commitdiff/6647248e3708843be93c7ca670cd219fe8e61026
  • Move deadlock and other interrupt handling in proc.c out of signal handlers. Deadlock checking was performed inside signal handlers up to now. While it's a remarkable feat to have made this work reliably, it's quite complex to understand why that is the case. Partially it worked due to the assumption that semaphores are signal safe - which is not actually documented to be the case for sysv semaphores. The reason we had to rely on performing this work inside signal handlers is that semaphores aren't guaranteed to be interruptable by signals on all platforms. But now that latches provide a somewhat similar API, which actually has the guarantee of being interruptible, we can avoid doing so. Signalling between ProcSleep, ProcWakeup, ProcWaitForSignal and ProcSendSignal is now done using latches. This increases the likelihood of spurious wakeups. As spurious wakeup already were possible and aren't likely to be frequent enough to be an actual problem, this seems acceptable. This change would allow for further simplification of the deadlock checking, now that it doesn't have to run in a signal handler. But even if I were motivated to do so right now, it would still be better to do that separately. Such a cleanup shouldn't have to be reviewed a the same time as the more fundamental changes in this commit. There is one possible usability regression due to this commit. Namely it is more likely than before that log_lock_waits messages are output more than once. Reviewed-By: Heikki Linnakangas http://git.postgresql.org/pg/commitdiff/6753333f55e1d9bcb9da4323556b456583624a07
  • Remove the option to service interrupts during PGSemaphoreLock(). The remaining caller (lwlocks) doesn't need that facility, and we plan to remove ImmedidateInterruptOK entirely. That means that interrupts can't be serviced race-free and portably anyway, so there's little reason for keeping the feature. Reviewed-By: Heikki Linnakangas http://git.postgresql.org/pg/commitdiff/d06995710bc7e347d39866c1793ae282498d65e0
  • Remove remnants of ImmediateInterruptOK handling. Now that nothing sets ImmediateInterruptOK to true anymore, we can remove all the supporting code. Reviewed-By: Heikki Linnakangas http://git.postgresql.org/pg/commitdiff/2505ce0be0b686c5a0a5141e9d080e7fdc35988a
  • Remove ill-conceived Assertion in ProcessClientWriteInterrupt(). It's perfectly fine to have blocked interrupts when ProcessClientWriteInterrupt() is called. In fact it's commonly the case when emitting error reports. And we deal with that correctly. Even if that'd not be the case, it'd be a bad location for such a assertion. Because ProcessClientWriteInterrupt() is only called when the socket is blocked it's hard to hit. Per Heikki and buildfarm animals nightjar and dunlin. http://git.postgresql.org/pg/commitdiff/3a54f4a4944b849846c5a1494810ea06bd6601b7
  • Add missing float.h include to snprintf.c. On windows _isnan() (which isnan() is redirected to in port/win32.h) is declared in float.h, not math.h. Per buildfarm animal currawong. Backpatch to all supported branches. http://git.postgresql.org/pg/commitdiff/ff8ca3b04c15eace1de1a7a66e3a05426b07b773

Fujii Masao a poussé :

Michael Meskes a poussé :

Bruce Momjian a poussé :

  • to_char(): prevent writing beyond the allocated buffer. Previously very long localized month and weekday strings could overflow the allocated buffers, causing a server crash. Reported and patch reviewed by Noah Misch. Backpatch to all supported versions. Security: CVE-2015-0241 http://git.postgresql.org/pg/commitdiff/9241c84cbcb82174a7d986242f233aa6472c1129
  • to_char(): prevent accesses beyond the allocated buffer. Previously very long field masks for floats could access memory beyond the existing buffer allocated to hold the result. Reported by Andres Freund and Peter Geoghegan. Backpatch to all supported versions. Security: CVE-2015-0241 http://git.postgresql.org/pg/commitdiff/0150ab567bcf5e5913e2b62a1678f84cc272441f
  • port/snprintf(): fix overflow and do padding. Prevent port/snprintf() from overflowing its local fixed-size buffer and pad to the desired number of digits with zeros, even if the precision is beyond the ability of the native sprintf(). port/snprintf() is only used on systems that lack a native snprintf(). Reported by Bruce Momjian. Patch by Tom Lane. Backpatch to all supported versions. Security: CVE-2015-0242 http://git.postgresql.org/pg/commitdiff/29725b3db67ad3f09da1a7fb6690737d2f8d6c0a

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Kyotaro HORIGUCHI sent in another revision of a patch to make fetch_size settable per foreign server and foreign table.
  • Kyotaro HORIGUCHI sent in a patch to ensure pg_basebackup sends feedback at desired times.
  • Andres Freund sent in a WIP patch to cache snapshots in GetSnapshotData().
  • Ryan Kelly sent in two revisions of a patch to add LINE: hint when schemaname.typename is a non-existent schema.
  • Kyotaro HORIGUCHI sent in a patch to add regnamespace and regrole types.
  • Peter Geoghegan sent in a patch to Make UPDATE privileges distinct from INSERT privileges in RTEs.
  • Robert Haas sent in two more revisions of a patch to add parallel mode and parallel contexts.
  • Noah Misch sent in a patch to allow no-rewrite timestamp<->timestamptz conversions.
  • Michael Paquier sent in two more revisions of a patch to make log_autovacuum_min_duration a relation option.
  • Heikki Linnakangas sent in a patch to simplify the waiting logic in reading from / writing to client.
  • Ian Lawrence Barwick sent in a patch to ensure that paths are canonicalized in pg_basebackup.
  • Fujii Masao, Michael Paquier, and Syed Rahila traded patches to implement compression of full-page writes.
  • Michael Paquier sent in three more revisions of a patch to add wal_availability_check_interval.
  • Daniel Bausch sent in a PoC patch to prefetch from storage for indexscans.
  • Amit Kapila sent in another revision of a patch to allow doing sequential scans in parallel.
  • Robert Haas sent in a patch to support parallel heap scans, and another to implement contrib/parallel_count, now using heap_parallel_beginscan.
  • Stas Kelvich sent in a patch to add kNN support for the contrib cube extension.
  • Jeff Davis sent in a patch sketching out Inclusion Constraints, a generlized form of foreign key.
  • Andreas Karlsson sent in another revision of a patch to reduce the lock strength of trigger and foreign key DDL.
  • Robert Haas sent in a patch to assess parallel-safety of query plans before attempting to parallelize them.

par N Bougain le mercredi 18 février 2015 à 01h50

jeudi 5 février 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 25 janvier 2015

Inscriptions pour la PGConf.US ouvertes avec offre spéciale pour les lève-tôt : http://pgconfus2015.eventbrite.com/?aff=pgann1

[ndt : meetup du PUG parisien ce jeudi 5 : http://www.meetup.com/PostgreSQL-User-Group-Paris/events/220230052/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Robert Haas a poussé :

  • BRIN typo fix. Amit Langote http://git.postgresql.org/pg/commitdiff/9d54b93239040dab49111b6d7e9d6bfe9c71f419
  • Typo fix. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/1605291b6c14be92915948d17f5509191632c97f
  • Use abbreviated keys for faster sorting of text datums. This commit extends the SortSupport infrastructure to allow operator classes the option to provide abbreviated representations of Datums; in the case of text, we abbreviate by taking the first few characters of the strxfrm() blob. If the abbreviated comparison is insufficent to resolve the comparison, we fall back on the normal comparator. This can be much faster than the old way of doing sorting if the first few bytes of the string are usually sufficient to resolve the comparison. There is the potential for a performance regression if all of the strings to be sorted are identical for the first 8+ characters and differ only in later positions; therefore, the SortSupport machinery now provides an infrastructure to abort the use of abbreviation if it appears that abbreviation is producing comparatively few distinct keys. HyperLogLog, a streaming cardinality estimator, is included in this commit and used to make that determination for text. Peter Geoghegan, reviewed by me. http://git.postgresql.org/pg/commitdiff/4ea51cdfe85ceef8afabceb03c446574daa0ac23
  • Add strxfrm_l to list of functions where Windows adds an underscore. Per buildfarm failure on bowerbird after last night's commit 4ea51cdfe85ceef8afabceb03c446574daa0ac23. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/f32a1fa462c88676105445f26885d7a0656b6806
  • Disable abbreviated keys on Windows. Most of the Windows buildfarm members (bowerbird, hamerkop, currawong, jacana, brolga) are unhappy with yesterday's abbreviated keys patch, although there are some (narwhal, frogmouth) that seem OK with it. Since there's no obvious pattern to explain why some are working and others are failing, just disable this across-the-board on Windows for now. This is a bit unfortunate since the optimization will be a big win in some cases, but we can't leave the buildfarm broken. http://git.postgresql.org/pg/commitdiff/1be4eb1b2d436d1375899c74e4c74486890d8777
  • Heavily refactor btsortsupport_worker. Prior to commit 4ea51cdfe85ceef8afabceb03c446574daa0ac23, this function only had one job, which was to decide whether we could avoid trampolining through the fmgr layer when performing sort comparisons. As of that commit, it has a second job, which is to decide whether we can use abbreviated keys. Unfortunately, those two tasks are somewhat intertwined in the existing coding, which is likely why neither Peter Geoghegan nor I noticed prior to commit that this calls pg_newlocale_from_collation() in cases where it didn't previously. The buildfarm noticed, though. To fix, rewrite the logic so that the decision as to which comparator to use is more cleanly separated from the decision about abbreviation. http://git.postgresql.org/pg/commitdiff/b529b65d1bf8537ca7fa024760a9782d7c8b66e5
  • More fixes for abbreviated keys infrastructure. First, when LC_COLLATE = C, bttext_abbrev_convert should use memcpy() rather than strxfrm() to construct the abbreviated key, because the authoritative comparator uses memcpy(). If we do anything else here, we might get inconsistent answers, and the buildfarm says this risk is not theoretical. It should be faster this way, too. Second, while I'm looking at bttext_abbrev_convert, convert a needless use of goto into the loop it's trying to implement into an actual loop. Both of the above problems date to the original commit of abbreviated keys, commit 4ea51cdfe85ceef8afabceb03c446574daa0ac23. Third, fix a bogus assignment to tss->locale before tss is set up. That's a new goof in commit b529b65d1bf8537ca7fa024760a9782d7c8b66e5. http://git.postgresql.org/pg/commitdiff/b181a91981203f6ec9403115a2917bd3f9473707
  • Repair brain fade in commit b181a91981203f6ec9403115a2917bd3f9473707. The split between which things need to happen in the C-locale case and which needed to happen in the locale-aware case was a few bricks short of a load. Try to fix that. http://git.postgresql.org/pg/commitdiff/d060e07fa919e0eb681e2fa2cfbe63d6c40eb2cf
  • Add an explicit cast to Size to hyperloglog.c. MSVC generates a warning here; we hope this will make it happy. Report by Michael Paquier. Patch by David Rowley. http://git.postgresql.org/pg/commitdiff/6a3c6ba0ba2bf86254d43b40569cdf3012cf12aa
  • Don't use abbreviated keys for the final merge pass. When we write tuples out to disk and read them back in, the abbreviated keys become non-abbreviated, because the readtup routines don't know anything about abbreviation. But without this fix, the rest of the code still thinks the abbreviation-aware compartor should be used, so chaos ensues. Report by Andrew Gierth; patch by Peter Geoghegan. http://git.postgresql.org/pg/commitdiff/5cefbf5a6c4466ac6b1cc2a4316b4eba9108c802
  • Fix typos, update README. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/d1747571b63444615730ca17474e9e89974c48ad

Álvaro Herrera a poussé :

Andres Freund a poussé :

  • Fix various shortcomings of the new PrivateRefCount infrastructure. As noted by Tom Lane the improvements in 4b4b680c3d6 had the problem that in some situations we searched, entered and modified entries in the private refcount hash while holding a spinlock. I had tried to keep the logic entirely local to PinBuffer_Locked(), but that's not really possible given it's called with a spinlock held... Besides being disadvantageous from a performance point of view, this also has problems with error handling safety. If we failed inserting an entry into the hashtable due to an out of memory error, we'd error out with a held spinlock. Not good. Change the way private refcounts are manipulated: Before a buffer can be tracked an entry has to be reserved using ReservePrivateRefCountEntry(); then, if a entry is not found using GetPrivateRefCountEntry(), it can be entered with NewPrivateRefCountEntry(). Also take advantage of the fact that PinBuffer_Locked() currently is never called for buffers that already have been pinned by the current backend and don't search the private refcount entries for preexisting local pins. That results in a small, but measurable, performance improvement. Additionally make ReleaseBuffer() always call UnpinBuffer() for shared buffers. That avoids duplicating work in an eventual UnpinBuffer() call that already has been done in ReleaseBuffer() and also saves some code. Per discussion with Tom Lane. Discussion: 15028.1418772313@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/2d115e47c861878669ba0814b3d97a4e4c347e8b

Tom Lane a poussé :

  • Adjust "pgstat wait timeout" message to be a translatable LOG message. Per discussion, change the log level of this message to be LOG not WARNING. The main point of this change is to avoid causing buildfarm run failures when the stats collector is exceptionally slow to respond, which it not infrequently is on some of the smaller/slower buildfarm members. This change does lose notice to an interactive user when his stats query is looking at out-of-date stats, but the majority opinion (not necessarily that of yours truly) is that WARNING messages would probably not get noticed anyway on heavily loaded production systems. A LOG message at least ensures that the problem is recorded somewhere where bulk auditing for the issue is possible. Also, instead of an untranslated "pgstat wait timeout" message, provide a translatable and hopefully more understandable message "using stale statistics instead of current ones because stats collector is not responding". The original text was written hastily under the assumption that it would never really happen in practice, which we now know to be unduly optimistic. Back-patch to all active branches, since we've seen the buildfarm issue in all branches. http://git.postgresql.org/pg/commitdiff/75b48e1fff8a4dedd3ddd7b76f6360b5cc9bb741
  • In pg_regress, remove the temporary installation upon successful exit. This results in a very substantial reduction in disk space usage during "make check-world", since that sequence involves creation of numerous temporary installations. It should also help a bit in the buildfarm, even though the buildfarm script doesn't create as many temp installations, because the current script misses deleting some of them; and anyway it seems better to do this once in one place rather than expecting that script to get it right every time. In 9.4 and HEAD, also undo the unwise choice in commit b1aebbb6a86e96d7 to report strerror(errno) after a rmtree() failure. rmtree has already reported that, possibly for multiple failures with distinct errnos; and what's more, by the time it returns there is no good reason to assume that errno still reflects the last reportable error. So reporting errno here is at best redundant and at worst badly misleading. Back-patch to all supported branches, so that future revisions of the buildfarm script can rely on this behavior. http://git.postgresql.org/pg/commitdiff/aa719391d5537d30068109e1f96991d0e2c7dfc5
  • Prevent duplicate escape-string warnings when using pg_stat_statements. contrib/pg_stat_statements will sometimes run the core lexer a second time on submitted statements. Formerly, if you had standard_conforming_strings turned off, this led to sometimes getting two copies of any warnings enabled by escape_string_warning. While this is probably no longer a big deal in the field, it's a pain for regression testing. To fix, change the lexer so it doesn't consult the escape_string_warning GUC variable directly, but looks at a copy in the core_yy_extra_type state struct. Then, pg_stat_statements can change that copy to disable warnings while it's redoing the lexing. It seemed like a good idea to make this happen for all three of the GUCs consulted by the lexer, not just escape_string_warning. There's not an immediate use-case for callers to adjust the other two AFAIK, but making it possible is easy enough and seems like good future-proofing. Arguably this is a bug fix, but there doesn't seem to be enough interest to justify a back-patch. We'd not be able to back-patch exactly as-is anyway, for fear of breaking ABI compatibility of the struct. (We could perhaps back-patch the addition of only escape_string_warning by adding it at the end of the struct, where there's currently alignment padding space.) http://git.postgresql.org/pg/commitdiff/eb213acfe2a51ab3dc6d9bbe02d5d8a903366779
  • Remove no-longer-referenced src/port/gethostname.c. This file hasn't been part of any build since 2005, and even before that wasn't used unless you configured --with-krb4 (and had a machine without gethostname(2), obviously). What's more, we haven't actually called gethostname anywhere since then, either (except in thread_test.c, whose testing of this function is probably pointless). So we don't need it. http://git.postgresql.org/pg/commitdiff/9222cd84b0f227287f65df395d52dc7973a62d29
  • Replace a bunch more uses of strncpy() with safer coding. strncpy() has a well-deserved reputation for being unsafe, so make an effort to get rid of nearly all occurrences in HEAD. A large fraction of the remaining uses were passing length less than or equal to the known strlen() of the source, in which case no null-padding can occur and the behavior is equivalent to memcpy(), though doubtless slower and certainly harder to reason about. So just use memcpy() in these cases. In other cases, use either StrNCpy() or strlcpy() as appropriate (depending on whether padding to the full length of the destination buffer seems useful). I left a few strncpy() calls alone in the src/timezone/ code, to keep it in sync with upstream (the IANA tzcode distribution). There are also a few such calls in ecpg that could possibly do with more analysis. AFAICT, none of these changes are more than cosmetic, except for the four occurrences in fe-secure-openssl.c, which are in fact buggy: an overlength source leads to a non-null-terminated destination buffer and ensuing misbehavior. These don't seem like security issues, first because no stack clobber is possible and second because if your values of sslcert etc are coming from untrusted sources then you've got problems way worse than this. Still, it's undesirable to have unpredictable behavior for overlength inputs, so back-patch those four changes to all active branches. http://git.postgresql.org/pg/commitdiff/586dd5d6a5d59e406bc8032bb52625ffb904311c
  • Fix unsafe coding in ReorderBufferCommit(). "iterstate" must be marked volatile since it's changed inside the PG_TRY block and then used in the PG_CATCH stanza. Noted by Mark Wilding of Salesforce. (We really need to see if we can't get the C compiler to warn about this.) Also, reset iterstate to NULL after the mainline ReorderBufferIterTXNFinish call, to ensure the PG_CATCH block doesn't try to do that a second time. http://git.postgresql.org/pg/commitdiff/f8a4dd2e141a12e349882edecc683504acb82ec8
  • Clean up some mess in row-security patches. Fix unsafe coding around PG_TRY in RelationBuildRowSecurity: can't change a variable inside PG_TRY and then use it in PG_CATCH without marking it "volatile". In this case though it seems saner to avoid that by doing a single assignment before entering the TRY block. I started out just intending to fix that, but the more I looked at the row-security code the more distressed I got. This patch also fixes incorrect construction of the RowSecurityPolicy cache entries (there was not sufficient care taken to copy pass-by-ref data into the cache memory context) and a whole bunch of sloppiness around the definition and use of pg_policy.polcmd. You can't use nulls in that column because initdb will mark it NOT NULL --- and I see no particular reason why a null entry would be a good idea anyway, so changing initdb's behavior is not the right answer. The internal value of '\0' wouldn't be suitable in a "char" column either, so after a bit of thought I settled on using '*' to represent ALL. Chasing those changes down also revealed that somebody wasn't paying attention to what the underlying values of ACL_UPDATE_CHR etc really were, and there was a great deal of lackadaiscalness in the catalogs.sgml documentation for pg_policy and pg_policies too. This doesn't pretend to be a complete code review for the row-security stuff, it just fixes the things that were in my face while dealing with the bugs in RelationBuildRowSecurity. http://git.postgresql.org/pg/commitdiff/fd496129d160950ed681c1150ea8f627b292c511

Bruce Momjian a poussé :

Peter Eisentraut a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Etsuro Fujita sent in another revision of a patch to fix some odd behavior in EvalQualPlan in the PostgreSQL FDW.
  • Pavel Stehule sent in another revision of a patch to disallow the deprecated => operator.
  • Michael Paquier sent in a patch to install shared libraries in bin and lib with MSVC.
  • Dean Rasheed sent in two revisions of a patch to make RLS checks run before attempting to insert/update any data rather than afterwards.
  • Michael Paquier sent in two more revisions of a patch to add recovery_timeout option to control timeout of restore_command nonzero status code.
  • Michael Paquier sent in a patch to remove some useless checks for NULL pointers.
  • Stephen Frost sent in four more revisions of a patch to fix a column-privilege leak in error-message paths.
  • Gilles Darold sent in another revision of a patch to fix a bug in pg_dump which gets table dependencies in extensions wrong.
  • Andrew (RhodiumToad) Gierth sent in another revision of a patch to implement GROUPING SETS.
  • Kyotaro HORIGUCHI sent in two more revisions of a patch to implement asynchronous execution of postgres_fdw.
  • Peter Geoghegan sent in a patch to fix the spill-to-disk behavior for abbreviated keys.
  • Amit Kapila sent in two more revisions of a patch to implement parallel sequential scan.
  • Pavel Stehule sent in two more revisions of a patch to implement array_position().
  • Michael Paquier sent in a patch to clean up the tablefunc contrib extension.
  • Kyotaro HORIGUCHI sent in a PoC patch to implement limiting FETCH in CURSORs by the number of bytes.
  • Robert Haas sent in a patch to fix a collate failure reported by Kevin Grittner.
  • Pavel Stehule sent in another revision of a patch to implement assert in PL/pgsql.
  • Andreas Karlsson sent in two more revisions of a patch to reduce the lock strength of trigger and foreign key DDL.
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to implemetn TABLESAMPLE.
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to implement a sequence access method.
  • Andreas Karlsson sent in another revision of a patch to use 128-bit integers for sum, avg and statistics aggregates.
  • Peter Geoghegan sent in a patch to fix some minor issues with code comments related to abbreviated keys.
  • Bruce Momjian sent in two revisions of a patch to recommend using wal_log_hints to make hint bits on the standby match the master for pg_upgrade in the rsync case.
  • Álvaro Herrera sent in another revision of a patch to track TRUNCATE via pgstat.
  • Tomas Vondra sent in another revision of a patch to implement multivariate statistics.
  • Pavel Stehule sent in another revision of a patch to implement row_to_array.

par N Bougain le jeudi 5 février 2015 à 01h20

vendredi 23 janvier 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 18 janvier 2015

Giulio Calacoci, le principal développeur de Barman, accueillera un rassemblement "Découvrez Barman 1.4.0" à Prato, Italie, le 6 février prochain. Informations complémentaires et inscriptions : http://www.meetup.com/2ndQuadrant-Italia-PostgreSQL-Meetup/events/219801332/

L'appel à conférenciers pour la deuxième conférence suisse de PostgreSQL, devant se tenir à HSR Rapperswil les 25 & 26 juin 2015, est lancé avec, comme date limite, le 1er avril : http://www.postgres-conference.ch/cfp/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Stephen Frost a poussé :

  • Skip dead backends in MinimumActiveBackends. Back in ed0b409, PGPROC was split and moved to static variables in procarray.c, with procs in ProcArrayStruct replaced by an array of integers representing process numbers (pgprocnos), with -1 indicating a dead process which has yet to be removed. Access to procArray is generally done under ProcArrayLock and therefore most code does not have to concern itself with -1 entries. However, MinimumActiveBackends intentionally does not take ProcArrayLock, which means it has to be extra careful when accessing procArray. Prior to ed0b409, this was handled by checking for a NULL in the pointer array, but that check was no longer valid after the split. Coverity pointed out that the check could never happen and so it was removed in 5592eba. That didn't make anything worse, but it didn't fix the issue either. The correct fix is to check for pgprocno == -1 and skip over that entry if it is encountered. Back-patch to 9.2, since there can be attempts to access the arrays prior to their start otherwise. Note that the changes prior to 9.4 will look a bit different due to the change in 5592eba. Note that MinimumActiveBackends only returns a bool for heuristic purposes and any pre-array accesses are strictly read-only and so there is no security implication and the lack of fields complaints indicates it's very unlikely to run into issues due to this. Pointed out by Noah Misch. http://git.postgresql.org/pg/commitdiff/1bf4a84d0f9f7a442790d7948e96cd42eeb90a91

Tom Lane a poussé :

  • Use correct text domain for errcontext() appearing within ereport(). The mechanism added in commit dbdf9679d7d61b03a3bf73af9b095831b7010eb5 for associating the correct translation domain with errcontext strings potentially fails in cases where errcontext() is used within an ereport() macro. Such usage was not originally envisioned for errcontext(), but we do have a few places that do it. In this situation, the intended comma expression becomes just a couple of arguments to errfinish(), which the compiler might choose to evaluate right-to-left. Fortunately, in such cases the textdomain for the errcontext string must be the same as for the surrounding ereport. So we can fix this by letting errstart initialize context_domain along with domain; then it will have the correct value no matter which order the calls occur in. (Note that error stack callback functions are not invoked until errfinish, so normal usage of errcontext won't affect what happens for errcontext calls within the ereport macro.) In passing, make sure that errcontext calls within the main backend set context_domain to something non-NULL. This isn't a live bug because NULL would select the current textdomain() setting which should be the right thing anyway --- but it seems better to handle this completely consistently with the regular domain field. Per report from Dmitry Voronin. Backpatch to 9.3; before that, there wasn't any attempt to ensure that errcontext strings were translated in an appropriate domain. http://git.postgresql.org/pg/commitdiff/1f9bf05e539646103c518bcbb49c04919b238f7a
  • Avoid unexpected slowdown in vacuum regression test. I noticed the "vacuum" regression test taking really significantly longer than it used to on a slow machine. Investigation pointed the finger at commit e415b469b33ba328765e39fd62edcd28f30d9c3c, which added creation of an index using an extremely expensive index function. That function was evidently meant to be applied only twice ... but the test re-used an existing test table, which up till a couple lines before that had had over two thousand rows. Depending on timing of the concurrent regression tests, the intervening VACUUMs might have been unable to remove those recently-dead rows, and then the index build would need to create index entries for them too, leading to the wrap_do_analyze() function being executed 2000+ times not twice. Avoid this by using a different table that is guaranteed to have only the intended two rows in it. Back-patch to 9.0, like the commit that created the problem. http://git.postgresql.org/pg/commitdiff/5b3ce2c911a2ec9de13b0dea7e135ad945a14583
  • Fix some functions that were declared static then defined not-static. Per testing with a compiler that whines about this. http://git.postgresql.org/pg/commitdiff/7391e2513f486a5df3eebf132c6dd6c16cf4e1f1
  • Remove duplicate specification of -Ae for HP-UX C compiler. Autoconf has known about automatically selecting -Ae when needed for quite some time now, so remove the redundant addition in template/hpux. Noted while setting up buildfarm member pademelon. http://git.postgresql.org/pg/commitdiff/fd3d894e4ea0021efa2628e4dfc5fe0ed3071859
  • Allow CFLAGS from configure's environment to override automatic CFLAGS. Previously, configure would add any switches that it chose of its own accord to the end of the user-specified CFLAGS string. Since most compilers process these left-to-right, this meant that configure's choices would override the user-specified flags in case of conflicts. We'd rather that worked the other way around, so adjust the logic to put the user's string at the end not the beginning. There does not seem to be a need for a similar behavior change for CPPFLAGS or LDFLAGS: in those, the earlier switches tend to win (think -I or -L behavior) so putting the user's string at the front is fine. Backpatch to 9.4 but not earlier. I'm not planning to run buildfarm member guar on older branches, and it seems a bit risky to change this behavior in long-stable branches. http://git.postgresql.org/pg/commitdiff/85a2a8903f7e9151793308d0638621003aded5ae
  • Improve performance of EXPLAIN with large range tables. As of 9.3, ruleutils.c goes to some lengths to ensure that table and column aliases used in its output are unique. Of course this takes more time than was required before, which in itself isn't fatal. However, EXPLAIN was set up so that recalculation of the unique aliases was repeated for each subexpression printed in a plan. That results in O(N^2) time and memory consumption for large plan trees, which did not happen in older branches. Fortunately, the expensive work is the same across a whole plan tree, so there is no need to repeat it; we can do most of the initialization just once per query and re-use it for each subexpression. This buys back most (not all) of the performance loss since 9.2. We need an extra ExplainState field to hold the precalculated deparse context. That's no problem in HEAD, but in the back branches, expanding sizeof(ExplainState) seems risky because third-party extensions might have local variables of that struct type. So, in 9.4 and 9.3, introduce an auxiliary struct to keep sizeof(ExplainState) the same. We should refactor the APIs to avoid such local variables in future, but that's material for a separate HEAD-only commit. Per gripe from Alexey Bashtanov. Back-patch to 9.3 where the issue was introduced. http://git.postgresql.org/pg/commitdiff/a5cd70dcbc268381e13cb0b2973b5732856d186f
  • Rearrange explain.c's API so callers need not embed sizeof(ExplainState). The folly of the previous arrangement was just demonstrated: there's no convenient way to add fields to ExplainState without breaking ABI, even if callers have no need to touch those fields. Since we might well need to do that again someday in back branches, let's change things so that only explain.c has to have sizeof(ExplainState) compiled into it. This costs one extra palloc() per EXPLAIN operation, which is surely pretty negligible. http://git.postgresql.org/pg/commitdiff/8e166e164c7c4531d7eb150d836aa2357989237a
  • Fix use-of-already-freed-memory problem in EvalPlanQual processing. Up to now, the "child" executor state trees generated for EvalPlanQual rechecks have simply shared the ResultRelInfo arrays used for the original execution tree. However, this leads to dangling-pointer problems, because ExecInitModifyTable() is all too willing to scribble on some fields of the ResultRelInfo(s) even when it's being run in one of those child trees. This trashes those fields from the perspective of the parent tree, because even if the generated subtree is logically identical to what was in use in the parent, it's in a memory context that will go away when we're done with the child state tree. We do however want to share information in the direction from the parent down to the children; in particular, fields such as es_instrument *must* be shared or we'll lose the stats arising from execution of the children. So the simplest fix is to make a copy of the parent's ResultRelInfo array, but not copy any fields back at end of child execution. Per report from Manuel Kniep. The added isolation test is based on his example. In an unpatched memory-clobber-enabled build it will reliably fail with "ctid is NULL" errors in all branches back to 9.1, as a consequence of junkfilter->jf_junkAttNo being overwritten with $7f7f. This test cannot be run as-is before that for lack of WITH syntax; but I have no doubt that some variant of this problem can arise in older branches, so apply the code change all the way back. http://git.postgresql.org/pg/commitdiff/c480cb9d246cec5e1dd7d72956e792df16e5445d
  • Improve new caching logic in tbm_add_tuples(). For no significant extra complexity, we can cache knowledge that the target page is lossy, and save a hash_search per iteration in that case as well. This probably makes little difference, since the extra rechecks that must occur when pages are lossy are way more expensive than anything we can save here ... but we might as well do it if we're going to cache anything. http://git.postgresql.org/pg/commitdiff/779fdcdeeeb9cdbfd271f8dc5bde76ed0c7b0813
  • Show sort ordering options in EXPLAIN output. Up to now, EXPLAIN has contented itself with printing the sort expressions in a Sort or Merge Append plan node. This patch improves that by annotating the sort keys with COLLATE, DESC, USING, and/or NULLS FIRST/LAST whenever nondefault sort ordering options are used. The output is now a reasonably close approximation of an ORDER BY clause equivalent to the plan's ordering. Marius Timmer, Lukas Kreft, and Arne Scheffer; reviewed by Mike Blackwell. Some additional hacking by me. http://git.postgresql.org/pg/commitdiff/20af53d7191f84d0f5b86da4362e481b7e85d52a
  • Fix ancient thinko in default table rowcount estimation. The code used sizeof(ItemPointerData) where sizeof(ItemIdData) is correct, since we're trying to account for a tuple's line pointer. Spotted by Tomonari Katsumata (bug #12584). Although this mistake is of very long standing, no back-patch, since it's a relatively harmless error and changing it would risk changing default planner behavior in stable branches. (I don't see any change in regression test outputs here, but the buildfarm may think differently.) http://git.postgresql.org/pg/commitdiff/75df6dc083f7a989697b5002a421fb204f2eeddb

Alvaro Herrera a poussé :

Heikki Linnakangas a poussé :

  • Fix typos in comment. Plus some tiny wordsmithing of not-quite-typos. http://git.postgresql.org/pg/commitdiff/3dfce37627b76e4da9e1d6090beedb608cefafcb
  • Silence Coverity warnings about unused return values from pushJsonbValue(). Similar warnings from backend were silenced earlier by commit c8315930, but there were a few more contrib/hstore. Michael Paquier http://git.postgresql.org/pg/commitdiff/e37d474f91c3a8a88be28a65389c948a55f18075
  • Fix thinko in re-setting wal_log_hints flag from a parameter-change record. The flag is supposed to be copied from the record. Same issue with track_commit_timestamps, but that's master-only. Report and fix by Petr Jalinek. Backpatch to 9.4, where wal_log_hints was added. http://git.postgresql.org/pg/commitdiff/49b04188f83fb8cacf925978989bc20399e76786
  • Another attempt at fixing Windows Norwegian locale. Previous fix mapped "Norwegian (BokmÃ¥l)" locale, which contains a non-ASCII character, to the pure ASCII alias "norwegian-bokmal". However, it turns out that more recent versions of the CRT library, in particular MSVCR110 (Visual Studio 2012), changed the behaviour of setlocale() so that if you pass "norwegian-bokmal" to setlocale, it returns "Norwegian_Norway". That meant trouble, when setlocale(..., NULL) first returned "Norwegian (BokmÃ¥l)_Norway", which we mapped to "norwegian-bokmal_Norway", but another call to setlocale(..., "norwegian-bokmal_Norway") returned "Norwegian_Norway". That caused PostgreSQL to think that they are different locales, and therefore not compatible. That caused initdb to fail at CREATE DATABASE. Older CRT versions seem to accept "Norwegian_Norway" too, so change the mapping to return "Norwegian_Norway" instead of "norwegian-bokmal". Backpatch to 9.2 like the previous attempt. We haven't made a release that includes the previous fix yet, so we don't need to worry about changing the locale of existing clusters from "norwegian-bokmal" to "Norwegian_Norway". (Doing any mapping like this at all requires changing the locale of existing databases; the release notes need to include instructions for that). http://git.postgresql.org/pg/commitdiff/aa1d2fc5e91e396bec5bf8a8d10b6cc4af0b0fff
  • Advance backend's advertised xmin more aggressively. Currently, a backend will reset it's PGXACT->xmin value when it doesn't have any registered snapshots left. That covered the common case that a transaction in read committed mode runs several queries, one after each other, as there would be no snapshots active between those queries. However, if you hold cursors across each of the query, we didn't get a chance to reset xmin. To make that better, keep all the registered snapshots in a pairing heap, ordered by xmin so that it's always quick to find the snapshot with the smallest xmin. That allows us to advance PGXACT->xmin whenever the oldest snapshot is deregistered, even if there are others still active. Per discussion originally started by Jeff Davis back in 2009 and more recently by Robert Haas. http://git.postgresql.org/pg/commitdiff/94028691609f8e148bd4ce72c46163f018832a5b

Andres Freund a poussé :

  • Allow latches to wait for socket writability without waiting for readability. So far WaitLatchOrSocket() required to pass in WL_SOCKET_READABLE as that solely was used to indicate error conditions, like EOF. Waiting for WL_SOCKET_WRITEABLE would have meant to busy wait upon socket errors. Adjust the API to signal errors by returning the socket as readable, writable or both, depending on WL_SOCKET_READABLE/WL_SOCKET_WRITEABLE being specified. It would arguably be nicer to return WL_SOCKET_ERROR but that's not possible on platforms and would probably also result in more complex callsites. This previously had explicitly been forbidden in e42a21b9e6c9, as there was no strong use case at that point. We now are looking into making FE/BE communication use latches, so changing this makes sense. There also are some portability concerns because there cases of older platforms where select(2) is known to, in violation of POSIX, not return a socket as writable after the peer has closed it. So far the platforms where that's the case provide a working poll(2). If we find one where that's not the case, we'll need to add a workaround for that platform. Discussion: 20140927191243.GD5423@alap3.anarazel.de Reviewed-By: Heikki Linnakangas, Noah Misch http://git.postgresql.org/pg/commitdiff/4bad60e3fd9a5fc6070fd4d1bd820a280e174654
  • Add barriers to the latch code. Since their introduction latches have required barriers in SetLatch and ResetLatch - but when they were introduced there wasn't any barrier abstraction. Instead latches were documented to rely on the callsites to provide barrier semantics. Now that the barrier support looks halfway complete, add the necessary barriers to both latch implementations. Also remove a now superflous lock acquisition from syncrep.c and a superflous (and insufficient) barrier from freelist.c. There might be other cases that can now be simplified, but those are the only ones I've seen on a quick scan. We might want to backpatch this at some later point, but right now the barrier infrastructure in the backbranches isn't totally on par with master. Discussion: 20150112154026.GB2092@awork2.anarazel.de http://git.postgresql.org/pg/commitdiff/14e8803f101a54d99600683543b0f893a2e3f529
  • Remove some dead IsUnderPostmaster code from bootstrap.c. Since commit 626eb021988a2 has introduced the auxiliary process infrastructure, bootstrap_signals() was never used when forked from postmaster. Remove the IsUnderPostmaster specific code, and add a appropriate assertion. http://git.postgresql.org/pg/commitdiff/0139dea8f1cea49f13c22a3f645dbdd02b90d25c
  • Commonalize process startup code. Move common code, that was duplicated in every postmaster child/every standalone process, into two functions in miscinit.c. Not only does that already result in a fair amount of net code reduction but it also makes it much easier to remove more duplication in the future. The prime motivation wasn't code deduplication though, but easier addition of new common code. http://git.postgresql.org/pg/commitdiff/31c453165b5a656044ce1dbce89f5828c1c7e23c
  • Make logging_collector=on work with non-windows EXEC_BACKEND again. Commit b94ce6e80 reordered postmaster's startup sequence so that the tempfile directory is only cleaned up after all the necessary state for pg_ctl is collected. Unfortunately the chosen location is after the syslogger has been started; which normally is fine, except for !WIN32 EXEC_BACKEND builds, which pass information to children via files in the temp directory. Move the call to RemovePgTempFiles() to just before the syslogger has started. That's the first child we fork. Luckily EXEC_BACKEND is pretty much only used by endusers on windows, which has a separate method to pass information to children. That means the real world impact of this bug is very small. Discussion: 20150113182344.GF12272@alap3.anarazel.de Backpatch to 9.1, just as the previous commit was. http://git.postgresql.org/pg/commitdiff/2be82dcf17a18511df5153bcafe67a9c1387be1e
  • Add a default local latch for use in signal handlers. To do so, move InitializeLatchSupport() into the new common process initialization functions, and add a new global variable MyLatch. MyLatch is usable as soon InitPostmasterChild() has been called (i.e. very early during startup). Initially it points to a process local latch that exists in all processes. InitProcess/InitAuxiliaryProcess then replaces that local latch with PGPROC->procLatch. During shutdown the reverse happens. This is primarily advantageous for two reasons: For one it simplifies dealing with the shared process latch, especially in signal handlers, because instead of having to check for MyProc, MyLatch can be used unconditionally. For another, a later patch that makes FEs/BE communication use latches, now can rely on the existence of a latch, even before having gone through InitProcess. Discussion: 20140927191243.GD5423@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/59f71a0d0b56b2df48db4bf1738aece5551f7a47
  • Blindly try to fix a warning in s_lock.h when compiling with gcc on HPPA. The possibly, depending on compiler settings, generated warning was "warning: `S_UNLOCK' redefined". The hppa spinlock implementation doesn't follow the rules of s_lock.h and provides a gcc specific implementation outside of the the part of the file that's supposed to do that. It does so to avoid duplication between the HP compiler and gcc. That unfortunately means that S_UNLOCK is already defined when the HPPA specific section is reached. Undefine the generic fallback S_UNLOCK definition inside the HPPA section. That's far from pretty, but has the big advantage of being simple. If somebody is interested to fix this in a prettier way... This presumably got broken in the course of 0709b7ee72. Discussion: 20150114225919.GY5245@awork2.anarazel.de Per complaint from Tom Lane. http://git.postgresql.org/pg/commitdiff/6cfd5086e140b365086d61f25c519d046dfcf7f0
  • Make tbm_add_tuples more efficient by caching the last acccessed page. When adding a large number of tuples to a TID bitmap using tbm_add_tuples() sometimes a lot of time was spent looking up a page's entry in the bitmap's internal hashtable. Improve efficiency by caching the last accessed page, while iterating over the passed in tuples, hoping consecutive tuples will often be on the same page. In many cases that's a good bet, and in the rest the added overhead isn't big. Discussion: 54479A85.8060309@sigaev.ru Author: Teodor Sigaev Reviewed-By: David Rowley http://git.postgresql.org/pg/commitdiff/f5ae3ba4828ece02bae2d16b4cbce847fbcea850
  • Replace walsender's latch with the general shared latch. Relying on the normal shared latch simplifies interrupt/signal handling because we can rely on all signal handlers setting the proc latch. That in turn allows us to avoid the use of ImmediateInterruptOK, which arguably isn't correct because WaitLatchOrSocket isn't declared to be immediately interruptible. Also change sections that wait on the walsender's latch to notice interrupts quicker/more reliably and make them more consistent with each other. This is part of a larger "get rid of ImmediateInterruptOK" series. Discussion: 20150115020335.GZ5245@awork2.anarazel.de http://git.postgresql.org/pg/commitdiff/ff44fba46c09c37dd9e60da1cb0b3a9339eb085f
  • Fix use of already freed memory when dumping a database's security label. pg_dump.c:dumDatabase() called ArchiveEntry() with the results of a a query that was PQclear()ed a couple lines earlier. Backpatch to 9.2 where security labels for shared objects where introduced. http://git.postgresql.org/pg/commitdiff/525b84c576e119de7f2b0d00e3a99d559771aa7b

Robert Haas a poussé :

Noah Misch a poussé :

Peter Eisentraut a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Michael Paquier sent in three revisions of a patch to remove unused variables in hstore_to_jsonb.
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to implement a sequence access method.
  • Dean Rasheed sent in three more revisions of a patch to fix some infelicities between INSERT...ON CONFLICT and RLS.
  • Michael Paquier sent in two more revisions of a patch to implement table-level log_autovacuum_min_duration.
  • Michael Paquier sent in two revisions of a patch to create non-erroring memory allocation functions.
  • Michael Paquier sent in a patch to fix a memory leak in receivelog.c when receiving stream.
  • Ali Akbar sent in a patch to clarify documentation that made generate_series(numeric, numeric) harder to implement.
  • Oskari Saarenmaa sent in a patch to defines custom macros for each attribute and enables them individually for compilers that support them and never defines __attribute__.
  • Andreas Karlsson and Michael Paquier traded patches to reduce the needed lock strength of triggers and foreign key DDL.
  • Marco Nenciarini sent in two more revisions of a patch to implement incremental backup.
  • Kyotaro HORIGUCHI sent in three more revisions of a patch to allow async execution of the PostgreSQL FDW.
  • Tomas Vondra sent in another revision of a patch to reduce the amount of memory consumed by array_agg().
  • Michael Paquier sent in a patch to make mere absence of destination folders not be a failure mode.
  • Etsuro Fujita sent in a patch to fix an issue where EvalPlanQual behaves oddly for FDW queries involving system columns.
  • Heikki Linnakangas sent in two more revisions of a patch to add pg_rewind.
  • Adam Brightwell sent in a patch to add some additional role attributes.
  • Amit Langote sent in a patch to fix a typo in brin.c.
  • Dilip Kumar sent in another revision of a patch to allow vacuumdb to work in parallel.
  • Gilles Darold sent in a patch to fix an issue in pg_dump with unusally-named tables.
  • Robert Haas sent in another revision of a patch to implement parallel mode and parallel contexts.
  • Michael Paquier sent in two revisions of a patch to fix a situation where the error check always bypassed in tablefunc.c.
  • Alexander Korotkov sent in two more revisions of a patch to implement fillfactor for GIN indexes.
  • Andres Freund sent in a patch to fix various shortcomings of the new PrivateRefCount infrastructure.
  • Pavel Stehule sent in a patch to add an array_position() function to PL/pgsql.
  • Andres Freund and Michael Paquier traded patches to move binaries from contrib/ to bin/
  • Pavel Stehule sent in a patch to complete the deprecation of => for other than pair-like functionality.
  • Tom Lane sent in a patch to deal with pg_stat wait timeout.
  • Tom Lane sent in a patch to have the buildfarm clean up temporary installs when done.
  • Michael Paquier sent in another revision of a patch to add recovery_timeout option to control timeout of restore_command nonzero status code.

par N Bougain le vendredi 23 janvier 2015 à 01h53

Nouvelles hebdomadaires de PostgreSQL - 11 janvier 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Fujii Masao a poussé :

Andres Freund a poussé :

  • Fix oversight in recent pg_basebackup fix causing pg_receivexlog failures. A oversight in 2c0a485896 causes 'could not create archive status file "...": No such file or directory' errors in pg_receivexlog if the target directory doesn't happen to contain a arch aAS FM SM tatus directory. That's due to a stupidly left over 'true' constant instead of mark_done being passed down to ProcessXLogDataMsg(). The bug is only present in the master branch, and luckily wasn't released. Spotted by Fujii Masao. http://git.postgresql.org/pg/commitdiff/3c9e4cdbf2ec876dbb71aa80a4cc18662fbd41c2
  • Correctly handle relcache invalidation corner case during logical decoding. When using a historic snapshot for logical decoding it can validly happen that a relation that's in the relcache isn't visible to that historic snapshot. E.g. if a newly created relation is referenced in the query that uses the SQL interface for logical decoding and a sinval reset occurs. The earlier commit that fixed the error handling for that corner case already improves the situation as a ERROR is better than hitting an assertion... But it's obviously not good enough. So additionally allow that case without an error if a historic snapshot is set up - that won't allow an invalid entry to stay in the cache because it's a) already marked invalid and will thus be rebuilt during the next access b) the syscaches will be reset at the end of decoding. There might be prettier solutions to handle this case, but all that we could think of so far end up being much more complex than this quite simple fix. This fixes the assertion failures reported by the buildfarm (markhor, tick, leech) after the introduction of new regression tests in 89fd41b390a4. The failure there weren't actually directly caused by CLOBBER_CACHE_ALWAYS but the extraordinary long runtimes due to it lead to sinval resets triggering the behaviour. Discussion: 22459.1418656530@sss.pgh.pa.us Backpatch to 9.4 where logical decoding was introduced. http://git.postgresql.org/pg/commitdiff/3fabed070530271771c2345dd7f60b0b28f0aae3
  • Improve relcache invalidation handling of currently invisible relations. The corner case where a relcache invalidation tried to rebuild the entry for a referenced relation but couldn't find it in the catalog wasn't correct. The code tried to RelationCacheDelete/RelationDestroyRelation the entry. That didn't work when assertions are enabled because the latter contains an assertion ensuring the refcount is zero. It's also more generally a bad idea, because by virtue of being referenced somebody might actually look at the entry, which is possible if the error is trapped and handled via a subtransaction abort. Instead just error out, without deleting the entry. As the entry is marked invalid, the worst that can happen is that the invalid (and at some point unused) entry lingers in the relcache. Discussion: 22459.1418656530@sss.pgh.pa.us There should be no way to hit this case < 9.4 where logical decoding introduced a bug that can hit this. But since the code for handling the corner case is there it should do something halfway sane, so backpatch all the the way back. The logical decoding bug will be handled in a separate commit. http://git.postgresql.org/pg/commitdiff/31912d01d8d860ffaafb386437d149e1749962ab
  • Remove comment that was intended to have been removed before commit. Noticed by Amit Kapila http://git.postgresql.org/pg/commitdiff/f454144a34c2288418292ae7b01495724cfda359
  • Move comment about sun cc's __machine_rw_barrier being a full barrier. I'd accidentally written the comment besides the read barrier, instead of the full barrier, implementation. Noticed by Oskari Saarenmaa http://git.postgresql.org/pg/commitdiff/93be0950070b5bcb22600cb1cbb62f93c058a5dd
  • Fix logging of pages skipped due to pins during vacuum. The new logging introduced in 35192f06 made the incorrect assumption that scan_all vacuums would always wait for buffer pins; but they only do so if the page actually needs to be frozen. Fix that inaccuracy by removing the difference in log output based on scan_all and just always remove the same message. I chose to keep the split log message from the original commit for now, it seems likely that it'll be of use in the future. Also merge the line about buffer pins in autovacuum's log output into the existing "pages: ..." line. It seems odd to have a separate line about pins, without the "topic: " prefix others have. Also rename the new 'pinned_pages' variable to 'pinskipped_pages' because it actually tracks the number of pages that could *not* be pinned. Discussion: 20150104005324.GC9626@awork2.anarazel.de http://git.postgresql.org/pg/commitdiff/17eaae9897781c6dcc622b9327899329e4d2a4df
  • Fix alignment of pg_atomic_uint64 variables on some 32bit platforms. I failed to recognize that pg_atomic_uint64 wasn't guaranteed to be 8 byte aligned on some 32bit platforms - which it has to be on some platforms to guarantee the desired atomicity and which we assert. As this is all compiler specific code anyway we can just rely on compiler specific tricks to enforce alignment. I've been unable to find concrete documentation about the version that introduce the sunpro alignment support, so that might need additional guards. I've verified that this works with gcc x86 32bit, but I don't have access to any other 32bit environment. Discussion: op.xpsjdkil0sbe7t@vld-kuci. Per report from Vladimir Koković. http://git.postgresql.org/pg/commitdiff/db4ec2ffce3549225619cae51d828224a11a165f
  • Provide a generic fallback for pg_compiler_barrier using an extern function. If the compiler/arch combination does not provide compiler barriers, provide a fallback. That fallback simply consists out of a function call into a externally defined function. That should guarantee compiler barrierer semantics except for compilers that do inter translation unit/global optimization - those better provide an actual compiler barrier. Hopefully this fixes Tom's report of linker failures due to pg_compiler_barrier_impl not being provided. I'm not backpatching this commit as it builds on the new atomics infrastructure. If we decide an equivalent fix needs to be backpatched, I'll do so in a separate commit. Discussion: 27746.1420930690@sss.pgh.pa.us. Per report from Tom Lane. http://git.postgresql.org/pg/commitdiff/de6429a8fdd3538e977b482d90389785d733e373

Tom Lane a poussé :

  • Fix broken pg_dump code for dumping comments on event triggers. This never worked, I think. Per report from Marc Munro. In passing, fix funny spacing in the COMMENT ON command as a result of excess space in the "label" string. http://git.postgresql.org/pg/commitdiff/adfc157dd91471e995f3aa72e0fb5b50ecd93575
  • Fix libpq's behavior when /etc/passwd isn't readable. Some users run their applications in chroot environments that lack an /etc/passwd file. This means that the current UID's user name and home directory are not obtainable. libpq used to be all right with that, so long as the database role name to use was specified explicitly. But commit a4c8f14364c27508233f8a31ac4b10a4c90235a9 broke such cases by causing any failure of pg_fe_getauthname() to be treated as a hard error. In any case it did little to advance its nominal goal of causing errors in pg_fe_getauthname() to be reported better. So revert that and instead put some real error-reporting code in place. This requires changes to the APIs of pg_fe_getauthname() and pqGetpwuid(), since the latter had departed from the POSIX-specified API of getpwuid_r() in a way that made it impossible to distinguish actual lookup errors from "no such user". To allow such failures to be reported, while not failing if the caller supplies a role name, add a second call of pg_fe_getauthname() in connectOptions2(). This is a tad ugly, and could perhaps be avoided with some refactoring of PQsetdbLogin(), but I'll leave that idea for later. (Note that the complained-of misbehavior only occurs in PQsetdbLogin, not when using the PQconnect functions, because in the latter we will never bother to call pg_fe_getauthname() if the user gives a role name.) In passing also clean up the Windows-side usage of GetUserName(): the recommended buffer size is 257 bytes, the passed buffer length should be the buffer size not buffer size less 1, and any error is reported by GetLastError() not errno. Per report from Christoph Berg. Back-patch to 9.4 where the chroot failure case was introduced. The generally poor reporting of errors here is of very long standing, of course, but given the lack of field complaints about it we won't risk changing these APIs further back (even though they're theoretically internal to libpq). http://git.postgresql.org/pg/commitdiff/080eabe2e8a184ff40b7380aaaa9418714acace9
  • Remove configure test for nonstandard variants of getpwuid_r(). We had code that supposed that some platforms might offer a nonstandard version of getpwuid_r() with only four arguments. However, the 5-argument definition has been standardized at least since the Single Unix Spec v2, which is our normal reference for what's portable across all Unix-oid platforms. (What's more, this wasn't the only pre-standardization version of getpwuid_r(); my old HPUX 10.20 box has still another signature.) So let's just get rid of the now-useless configure step. http://git.postgresql.org/pg/commitdiff/8883bae33b55a52105b1b58d0e42c5a6bda09627
  • Fix portability breakage in pg_dump. Commit 0eea8047bf0e15b402b951e383e39236bdfe57d5 introduced some overly optimistic assumptions about what could be in a local struct variable's initializer. (This might in fact be valid code according to C99, but I've got at least one pre-C99 compiler that falls over on those nonconstant address expressions.) There is no reason whatsoever for main()'s workspace to not be static, so revert long_options[] to a static and make the DumpOptions struct static as well. http://git.postgresql.org/pg/commitdiff/44096f1c6644a97c92da6c96dc1d8d4e200d4a38

Bruce Momjian a poussé :

Ãlvaro Herrera a poussé :

Peter Eisentraut a poussé :

  • Fix namespace handling in xpath function. Previously, the xml value resulting from an xpath query would not have namespace declarations if the namespace declarations were attached to an ancestor element in the input xml value. That means the output value was not correct XML. Fix that by running the result value through xmlCopyNode(), which produces the correct namespace declarations. Author: Ali Akbar <the.apaan@gmail.com> http://git.postgresql.org/pg/commitdiff/79af9a1d2668c9edc8171f03c39e7fed571eeb98

Heikki Linnakangas a poussé :

  • Don't open a WAL segment for writing at end of recovery. Since commit ba94518a, we used XLogFileOpen to open the next segment for writing, but if the end-of-recovery happens exactly at a segment boundary, the new segment might not exist yet. (Before ba94518a, XLogFileOpen was correct, because we would open the previous segment if the switch happened at the boundary.) Instead of trying to create it if necessary, it's simpler to not bother opening the segment at all. XLogWrite() will open or create it soon anyway, after writing the checkpoint or end-of-recovery record. Reported by Andres Freund. http://git.postgresql.org/pg/commitdiff/1e78d81e883d7987de8da559d985958335820cc8

Robert Haas a poussé :

Noah Misch a poussé :

  • On Darwin, refuse postmaster startup when multithreaded. The previous commit introduced its report at LOG level to avoid surprises at minor release upgrade time. Compel users deploying the next major release to also deploy the reported workaround. http://git.postgresql.org/pg/commitdiff/2048e5b88198213ae15c4b59ddbe124f1a5dc2b7
  • Always set the six locale category environment variables in main(). Typical server invocations already achieved that. Invalid locale settings in the initial postmaster environment interfered, as could malloc() failure. Setting "LC_MESSAGES=pt_BR.utf8 LC_ALL=invalid" in the postmaster environment will now choose C-locale messages, not Brazilian Portuguese messages. Most localized programs, including all PostgreSQL frontend executables, do likewise. Users are unlikely to observe changes involving locale categories other than LC_MESSAGES. CheckMyDatabase() ensures that we successfully set LC_COLLATE and LC_CTYPE; main() sets the remaining three categories to locale "C", which almost cannot fail. Back-patch to 9.0 (all supported versions). http://git.postgresql.org/pg/commitdiff/6fdba8ceb071a3512d5685f1cd4f971ab4d562d1
  • Reject ANALYZE commands during VACUUM FULL or another ANALYZE. vacuum()'s static variable handling makes it non-reentrant; an ensuing null pointer deference crashed the backend. Back-patch to 9.0 (all supported versions). http://git.postgresql.org/pg/commitdiff/e415b469b33ba328765e39fd62edcd28f30d9c3c
  • On Darwin, detect and report a multithreaded postmaster. Darwin --enable-nls builds use a substitute setlocale() that may start a thread. Buildfarm member orangutan experienced BackendList corruption on account of different postmaster threads executing signal handlers simultaneously. Furthermore, a multithreaded postmaster risks undefined behavior from sigprocmask() and fork(). Emit LOG messages about the problem and its workaround. Back-patch to 9.0 (all supported versions). http://git.postgresql.org/pg/commitdiff/894459e59ffa5c7fee297b246c17e1f72564db1d

Stephen Frost a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Michael Paquier sent in a patch to complain about OOM of XLOG reader allocation in logical decoding code.
  • Andres Freund sent in a patch to improve handling of relcache invalidations to currently invisible relations.
  • Petr (PJMODOS) Jelinek sent in two revisions of a patch to fix handlign of event triggers in pg_dump.
  • Michael Paquier sent in a patch to add a restore_command_retry_interval to control retries of restore_command.
  • Fabien COELHO sent in two more revisions of a patch to add modulo (%) to pg_bench.
  • Petr (PJMODOS) Jelinek sent in two more revisions of a patch to track commit timestamps.
  • Atri Sharma sent in a patch to add functionality to specify ORDER BY in CREATE FUNCTION for SRFs.
  • Heikki Linnakangas sent in three more revisions of a patch to add pg_rewind.
  • Michael Paquier sent in two more revisions of a patch to add compression of full-page writes.
  • Kaigai Kouhei sent in another revision of a patch to add ctid scans as an example of the custom scan interface.
  • Andrew (RhodiumToad) Gierth sent in another revision of a patch to add GROUPING SETS.
  • Andrew Dunstan sent in a patch to add min and max execute statement time in pg_stat_statement.
  • Petr (PJMODOS) Jelinek sent in a patch to bring XLOG_PARAMETER_CHANGE into line with the new handling of wal_log_hints.
  • Robert Haas sent in two more revisions of a patch to implement parallel mode and parallel contexts.
  • Peter Eisentraut sent in another revision of a patch to turn recovery.conf into GUCs.
  • Kaigai Kouhei sent in another revision of a patch to implement a custom and foreign join interface.
  • Marius Timmer sent in another revision of a patch to add sort order to EXPLAIN output.
  • Tomas Vondra sent in another revision of a patch to do better memory accounting for memory-bounded hashaggs, another to do the aforementioned hashaggs, and a third to change hashcontext to aggcontext where needed.
  • Michael Paquier and Alexander Korotkov traded patches to add a fillfactor for GIN indexes.
  • Marco Nenciarini sent in a patch to implement a LSN map, one of whose outcomes will likely be incremental backup.
  • Craig Ringer sent in a patch to make server_version_num a GUC_REPORT.
  • Dean Rasheed sent in a patch to fix the documentation for CREATE POLICY.
  • Aaron Botsis sent in another revision of a patch to allow parsing JSON with embedded double quotes.
  • Petr (PJMODOS) Jelinek sent in two more revisions of a patch to implement TABLESAMPLE.
  • David Fetter sent in three more revisions of a patch to fix psql's mis-handling of connect string and URI parameters with \c[onnect].
  • Tomas Vondra and Ali Akbar traded patches to decrease the memory needed for array_agg().
  • David Rowley sent in another revision of a patch to allow removing INNER JOINs under some circumstances.
  • Amit Kapila sent in another revision of a patch to enable parallel sequential scans.
  • Kyotaro HORIGUCHI sent in another revision of a patch to allow asynchronous execution of postgres_fdw.
  • Abhijit Menon-Sen sent in another revision of a patch set to do better CRCs, depending on availability.
  • Dean Rasheed sent in a patch to improve RLS qual pushdowns by checking whether leakage is significant in direct terms like row contents.
  • Bruce Momjian sent in a patch to make PQputCopyEnd adhere to its API contract.
  • Andres Freund sent in a flock of patches intended to fix an issue that originally manifested as inability to escape from a blocked send().
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT {UPDATE | IGNORE}.

par N Bougain le vendredi 23 janvier 2015 à 01h36

mercredi 21 janvier 2015

Guillaume Lelarge

PostgreSQL - Architecture et notions avancées

Et voilà, j'ai fini par le faire. J'ai fini par me convaincre d'écrire un livre complet sur PostgreSQL.

C'est quelque chose qui me trottait dans la tête depuis longtemps. Surtout depuis que Thomas Reiss m'avait montré le Concept Guide d'Oracle. Je m'étais dit à ce moment-là : mince, c'est ça qu'il manque à PostgreSQL. Mais bon, ça demande beaucoup de temps, beaucoup de travail. Je n'avais pas très envie de me jeter là-dedans, même si je savais que certains éditeurs cherchaient des auteurs pour des livres sur PostgreSQL.

J'ai eu la chance de connaître Patricia Montcorgé lors de la traduction du livre de Greg Smith sur les performances avec PostgreSQL. Si bien que, après avoir lu le livre de Brendan Gregg sur la recherche de performances (Systems Performance: Enterprise and the Cloud, excellent livre, à mettre entre toutes les mains), je lui ai proposé deux projets : la traduction de ce livre et l'écriture d'un livre sur PostgreSQL. Elle m'a appris qu'elle avait fondé sa propre maison d'édition, qu'elle ne s'occupait plus de traductions, et que le livre sur PostgreSQL pourrait l'intéresser. On s'est rencontré, et, après lui avoir expliqué plus en profondeur mon projet, elle a pu m'expliquer comment elle voyait le travail avec elle. J'ai trouvé que ça ressemblait beaucoup à un projet libre, avec une version beta, du git, du docbook, des mises à jour facile, etc. Bref, j'étais bien accroché.

Je travaille sur ce livre depuis avril 2014. On a déjà subi un gros retard à cause du chapitre sur les processus : étudier le code de chaque processus a été particulièrement long. Mais bon, on a maintenant un rythme de croisière intéressant. On s'est décidé sur un chapitre tous les 4 à 6 semaines.

La version beta est sortie aujourd'hui. Elle est disponible par module ou complète, uniquement en version électronique pour le moment (jusqu'à la version finale en fait). Les chapitres seront livrés au fur et à mesure de leur écriture. Un forum est disponible pour les lecteurs qui voudraient laisser des commentaires ou des demandes ou des corrections.

La version finale sera mise à jour pour intégrer les changements effectués par la version 9.5.

Bon, je retourne bosser sur le livre :)

Preneur de tout commentaire par mail sur guillaume@lelarge.info.

par Guillaume Lelarge le mercredi 21 janvier 2015 à 22h50

lundi 5 janvier 2015

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 4 janvier 2015

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Assorted minor fixes for psql metacommand docs. Document the long forms of \H \i \ir \o \p \r \w ... apparently, we have a long and dishonorable history of leaving out the unabbreviated names of psql backslash commands. Avoid saying "Unix shell"; we can just say "shell" with equal clarity, and not leave Windows users wondering whether the feature works for them. Improve consistency of documentation of \g \o \w metacommands. There's no reason to use slightly different wording or markup for each one. http://git.postgresql.org/pg/commitdiff/ae17897ce84e545fd5a042530055df728b2ca98f
  • Improve consistency of parsing of psql's magic variables. For simple boolean variables such as ON_ERROR_STOP, psql has for a long time recognized variant spellings of "on" and "off" (such as "1"/"0"), and it also made a point of warning you if you'd misspelled the setting. But these conveniences did not exist for other keyword-valued variables. In particular, though ECHO_HIDDEN and ON_ERROR_ROLLBACK include "on" and "off" as possible values, none of the alternative spellings for those were recognized; and to make matters worse the code would just silently assume "on" was meant for any unrecognized spelling. Several people have reported getting bitten by this, so let's fix it. In detail, this patch: 1. Allows all spellings recognized by ParseVariableBool() for ECHO_HIDDEN and ON_ERROR_ROLLBACK. 2. Reports a warning for unrecognized values for COMP_KEYWORD_CASE, ECHO, ECHO_HIDDEN, HISTCONTROL, ON_ERROR_ROLLBACK, and VERBOSITY. 3. Recognizes all values for all these variables case-insensitively; previously there was a mishmash of case-sensitive and case-insensitive behaviors. Back-patch to all supported branches. There is a small risk of breaking existing scripts that were accidentally failing to malfunction; but the consensus is that the chance of detecting real problems and preventing future mistakes outweighs this. http://git.postgresql.org/pg/commitdiff/28551797a46c0ada035c34cf9cf2bc90448e3c26
  • Print more information about getObjectIdentityParts() failures. This might help us debug what's happening on some buildfarm members. In passing, reduce the message from ereport to elog --- it doesn't seem like this should be a user-facing case, so not worth translating. http://git.postgresql.org/pg/commitdiff/a486841eb11517e4a344fdebaf370f425a171c85
  • Docs: improve descriptions of ISO week-numbering date features. Use the phraseology "ISO 8601 week-numbering year" in place of just "ISO year", and make related adjustments to other terminology. The point of this change is that it seems some people see "ISO year" and think "standard year", whereupon they're surprised when constructs like to_char(..., "IYYY-Michael Meskes-DD") produce nonsensical results. Perhaps hanging a few more adjectives on it will discourage them from jumping to false conclusions. I put in an explicit warning against that specific usage, too, though the main point is to discourage people who haven't read this far down the page. In passing fix some nearby markup and terminology inconsistencies. http://git.postgresql.org/pg/commitdiff/2f03ae69875ff27a9667a7ec4655434900b64e55
  • Don't run rowsecurity in parallel with other regression tests. The short-lived event trigger in the rowsecurity test causes irreproducible failures when the concurrent tests do something that the event trigger can't cope with. Per buildfarm. http://git.postgresql.org/pg/commitdiff/7161b082bd9fc51e67a1031ea9d0313e8a48286b
  • Treat negative values of recovery_min_apply_delay as having no effect. At one point in the development of this feature, it was claimed that allowing negative values would be useful to compensate for timezone differences between master and slave servers. That was based on a mistaken assumption that commit timestamps are recorded in local time; but of course they're in UTC. Nor is a negative apply delay likely to be a sane way of coping with server clock skew. However, the committed patch still treated negative delays as doing something, and the timezone misapprehension survived in the user documentation as well. If recovery_min_apply_delay were a proper GUC we'd just set the minimum allowed value to be zero; but for the moment it seems better to treat negative settings as if they were zero. In passing do some extra wordsmithing on the parameter's documentation, including correcting a second misstatement that the parameter affects processing of Restore Point records. Issue noted by Michael Paquier, who also provided the code patch; doc changes by me. Back-patch to 9.4 where the feature was introduced. http://git.postgresql.org/pg/commitdiff/d6657d2a107bee20d63ec2f776d873242f9355e9

Tatsuo Ishii a poussé :

Heikki Linnakangas a poussé :

  • Revert the GinMaxItemSize calculation so that we fit 3 tuples per page. Commit 36a35c55 changed the divisor from 3 to 6, for no apparent reason. Reducing GinMaxItemSize like that created a dump/reload hazard: loading a 9.3 database to 9.4 might fail with "index row size XXX exceeds maximum 1352 for index ..." error. Revert the change. While we're at it, make the calculation slightly more accurate. It used to divide the available space on page by three, then subtract sizeof(ItemIdData), and finally round down. That's not totally accurate; the item pointers for the three items are packed tight right after the page header, but there is alignment padding after the item pointers. Change the calculation to reflect that, like BTMaxItemSize does. I tested this with different block sizes on systems with 4- and 8-byte alignment, and the value after the final MAXALIGN_DOWN was the same with both methods on all configurations. So this does not make any difference currently, but let's be tidy. Also add a comment explaining what the macro does. This fixes bug #12292 reported by Robert Thaler. Backpatch to 9.4, where the bug was introduced. http://git.postgresql.org/pg/commitdiff/930fd68455576dada48521a6cef642caddea3c63

Álvaro Herrera a poussé :

  • Fix object_address expected output. Per pink buildfarm http://git.postgresql.org/pg/commitdiff/5b447ad3a98aa6f059ba467eb10a832eb1544985
  • Use TypeName to represent type names in certain commands. In COMMENT, DROP, SECURITY LABEL, and the new pg_get_object_address function, we were representing types as a list of names, same as other objects; but types are special objects that require their own representation to be totally accurate. In the original COMMENT code we had a note about fixing it which was lost in the course of c10575ff005. Change all those places to use TypeName instead, as suggested by that comment. Right now the original coding doesn't cause any bugs, so no backpatch. It is more problematic for proposed future code that operate with object addresses from the SQL interface; type details such as array-ness are lost when working with the degraded representation. Thanks to Petr Jelínek and Dimitri Fontaine for offlist help on finding a solution to a shift/reduce grammar conflict. http://git.postgresql.org/pg/commitdiff/3f88672a4e4d8e648d24ccc65937da61c7660854
  • Add pg_identify_object_as_address. This function returns object type and objname/objargs arrays, which can be passed to pg_get_object_address. This is especially useful because the textual representation can be copied to a remote server in order to obtain the corresponding OID-based address. In essence, this function is the inverse of recently added pg_get_object_address(). Catalog version bumped due to the addition of the new function. Also add docs to pg_get_object_address. http://git.postgresql.org/pg/commitdiff/a676201490c8113b4692562126c77a29dfd8dac1
  • pg_event_trigger_dropped_objects: Add name/args output columns. These columns can be passed to pg_get_object_address() and used to reconstruct the dropped objects identities in a remote server containing similar objects, so that the drop can be replicated. Reviewed by Stephen Frost, Heikki Linnakangas, Abhijit Menon-Sen, Andres Freund. http://git.postgresql.org/pg/commitdiff/72dd233d3ef12da0f041bb3024deaa6d2950369c
  • Add missing pstrdup calls. The one for the OCLASS_COLLATION case was noticed by CLOBBER_CACHE_ALWAYS buildfarm members; the others I spotted by manual code inspection. Also remove a redundant check. http://git.postgresql.org/pg/commitdiff/ba66c9d0681f0c5ec834255fb4e7f6ef6b626747
  • Fix thinko in lock mode enum Commit 0e5680f4737a9c6aa94aa9e77543e5de60411322 contained a thinko mixing LOCKMODE with LockTupleMode. This caused misbehavior in the case where a tuple is marked with a multixact with at most a FOR SHARE lock, and another transaction tries to acquire a FOR NO KEY EXCLUSIVE lock; this case should block but doesn't. Include a new isolation tester spec file to explicitely try all the tuple lock combinations; without the fix it shows the problem: starting permutation: s1_begin s1_lcksvpt s1_tuplock2 s2_tuplock3 s1_commit step s1_begin: BEGIN; step s1_lcksvpt: SELECT * FROM multixact_conflict FOR KEY SHARE; SAVEPOINT foo; a 1 step s1_tuplock2: SELECT * FROM multixact_conflict FOR SHARE; a 1 step s2_tuplock3: SELECT * FROM multixact_conflict FOR NO KEY UPDATE; a 1 step s1_commit: COMMIT; With the fixed code, step s2_tuplock3 blocks until session 1 commits, which is the correct behavior. All other cases behave correctly. Backpatch to 9.3, like the commit that introduced the problem. http://git.postgresql.org/pg/commitdiff/d5e3d1e969d2f65009f718d3100d6565f47f9112

Robert Haas a poussé :

Magnus Hagander a poussé :

Andres Freund a poussé :

  • Prevent WAL files created by pg_basebackup -x/X from being archived again. WAL (and timeline history) files created by pg_basebackup did not maintain the new base backup's archive status. That's currently not a problem if the new node is used as a standby - but if that node is promoted all still existing files can get archived again. With a high wal_keep_segment settings that can happen a significant time later - which is quite confusing. Change both the backend (for the -x/-X fetch case) and pg_basebackup (for -X stream) itself to always mark WAL/timeline files included in the base backup as .done. That's in line with walreceiver.c doing so. The verbosity of the pg_basebackup changes show pretty clearly that it needs some refactoring, but that'd result in not be backpatchable changes. Backpatch to 9.1 where pg_basebackup was introduced. Discussion: 20141205002854.GE21964@awork2.anarazel.de http://git.postgresql.org/pg/commitdiff/2c0a4858965f1375dadb45df9f3a054bfcb283f5
  • Add pg_string_endswith as the start of a string helper library in src/common. Backpatch to 9.3 where src/common was introduced, because a bugfix that needs to be backpatched, requires the function. Earlier branches will have to duplicate the code. http://git.postgresql.org/pg/commitdiff/ccb161b66addc9d0ede31359c05f7e9df61ab8d4
  • Fix inconsequential fd leak in the new mark_file_as_archived() function. As every error in mark_file_as_archived() will lead to a failure of pg_basebackup the FD leak couldn't ever lead to a real problem. It seems better to fix the leak anyway though, rather than silence Coverity, as the usage of the function might get extended or copied at some point in the future. Pointed out by Coverity. Backpatch to 9.2, like the relevant part of the previous patch. http://git.postgresql.org/pg/commitdiff/0398ece4c5baacc98f0fd46a731518b3ccb25bb3
  • Remove superflous variable from xlogreader's XLogFindNextRecord(). Pointed out by Coverity. Since this is mere, and debatable, cosmetics I'm not backpatching this. http://git.postgresql.org/pg/commitdiff/14570c28289f82030172c699ec877dd26d04940a
  • Add missing va_end() call to a early exit in dmetaphone.c's StringAt(). Pointed out by Coverity. Backpatch to all supported branches, the code has been that way for a long while. http://git.postgresql.org/pg/commitdiff/58bc4747be26f6849ca38e67e24abdd6aecba3b1
  • Fix off-by-one in pg_xlogdump's fuzzy_open_file(). In the unlikely case of stdin (fd 0) being closed, the off-by-one would lead to pg_xlogdump failing to open files. Spotted by Coverity. Backpatch to 9.3 where pg_xlogdump was introduced. http://git.postgresql.org/pg/commitdiff/d1c575230d49929f82c6d91e9b3070a9f4018718
  • Correctly handle test durations of more than 2147s in pg_test_timing. Previously the computation of the total test duration, measured in microseconds, accidentally overflowed due to accidentally using signed 32bit arithmetic. As the only consequence is that pg_test_timing invocations with such, overly large, durations never finished the practical consequences of this bug are minor. Pointed out by Coverity. Backpatch to 9.2 where pg_test_timing was added. http://git.postgresql.org/pg/commitdiff/8cadeb792cd08478a73cb0941bb67f03e8465090
  • Add error handling for failing fstat() calls in copy.c. These calls are pretty much guaranteed not to fail unless something has gone horribly wrong, and even in that case we'd just error out a short time later. But since several code checkers complain about the missing check it seems worthwile to fix it nonetheless. Pointed out by Coverity. http://git.postgresql.org/pg/commitdiff/2ea95959afa225118374ab1691a5ccf84ae05ce8

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Abhijit Menon-Sen sent in another revision of a patch to implement pg_audit.
  • Adam Brightwell sent in a patch to remove CATUPDATE.
  • Peter Geoghegan sent in two revisions of a patch to document bt_page_items()'s ctid field.
  • Abhijit Menon-Sen sent in three more revisions of a patch to implement more CRC algorithms, using more efficient ones when available.
  • Bruce Momjian sent in two patches to align BufferDescriptions correctly.
  • Álvaro Herrera sent in a patch to use TypeName to represent type names in certain commands.
  • Álvaro Herrera sent in a patch to add SQL-level callable function to obtain object type + identity, etc.
  • Alexei Vasiliev sent in two more revisions of a patch to add a recovery_timeout option to control the timeout of a restore_command nonzero status code.
  • David Fetter sent in a patch to fix service and URI requests in psql's \c[onnect] command.
  • David Rowley sent in a patch to improve performance for outer joins where outer side is unique.
  • Fabien COELHO sent in a patch to standardize some wording across the function documentation.
  • Fabien COELHO and David Rowley traded patches to add the modulo (%) operator to pgbench.
  • Amit Kapila sent in another revision of a patch to allow parallel cores to be used by vacuumdb.
  • Heikki Linnakangas sent in another revision of a patch to redesign checkpoint segments.
  • Noah Misch sent in a patch to try to track down a buildfarm failure in transaction isolation.
  • Ashutosh Bapat sent in a patch to attempt to use 2PC on transactions involving multiple foreign PostgreSQL servers.
  • Peter Geoghegan sent in two revisions of a patch to fix some issues in value locking.
  • David Fetter sent in a patch to fix the documentation for the default location of the system-wide pg_service configuration.

par N Bougain le lundi 5 janvier 2015 à 19h39

jeudi 1 janvier 2015

ulhume

Rooter le LG G Pad 7.0

Lorsque l&aposon achète un device Android, se pose toujours le même problème du &aposcomment le rooter&apos. Voici donc comment faire pour la toute nouvelle tablette LG G Pad 6.0.

Pourquoi faire ?

Comme mon sujet pour l&aposannée 2015 va clairement être le développement d&aposapplications mobile, après m&aposêtre commis à acheter un Mac Mini et un iPad Mini, j&aposai aussi fait l&aposacquisition d&aposun fairphone et d&aposune tablette Android, le LG G Pad 7.0.

Bref, tout cela est bien beau, mais pour bosser avec Android, rooter est l&aposétape nécessaire. C&aposest déjà pour moi une nécessité philosophique : l&aposidée d&aposavoir payé un engin dont je ne peux pas faire ce que je veux m&aposest basiquement insupportable. Mais au delà de mes états d&aposâme, l&aposaccès root permet de faire des choses indispensables comme, par exemple, utiliser ADB en mode réseau (c&aposest à dire sans cable USB). Cela permet aussi d&aposinstaller d&aposautres versions d&aposAndroid, ce qui n&aposest pas forcement nécessaire tout de suite, mais qui le deviendra inéluctablement lorsque le constructeur aura décrété que votre appareil est en "fin de vie" et qu&aposil ne maintiendra plus son système d&aposexploitation.

Côté fairphone, c&aposest du velour. En effet ce téléphone ne se contente pas d&aposêtre humainement éthique, il l&aposest aussi techniquement. Entendez par là qu&aposil est fournit avec un manuel de démontage, un catalogue de pièces détachées abordables et surtout il est rooté en standard. Que du bonheur donc.

Pour ce qui est du LG c&aposest une autre paire de manche. Ce constructeur, comme beaucoup d&aposautres, ne considère en effet pas normal d&aposavoir le droit d&aposaccéder totalement à sa machine. Voyons comment y remédier. À noter que cette technique semble être la même pour les autres tablettes de la série (8.0 et 10.3) ainsi que pour les smartphones LG.

Notez aussi que tout ce qui suit est fait à vos risques et périls. Tout ce qui suit est de nature à annuler votre garantie et je ne me considérerais pas responsable si vous bloquez votre machine.

Passer en mode développeur

La première étape consiste déjà à activer le mode développeur, ce qui n&aposest pas aussi évident que l&aposon pourrait se l&aposimaginer. Pour cela vous devrez aller dans les réglages, puis dans le menu à propos de la tablette, et enfin dans information sur le logiciel. Arrivé là, accrochez-vous bien, vous devez taper plusieurs fois sur le Numéro de build. La tablette vous demande alors si vous êtes certain de votre action, puis fera apparaître, à la positive, le menu tant utile pour les développeurs.

Dans ce menu, vous allez devoir vous rendre dans la section debuggage pour activer le debuggage de la connection USB. Ce qui est le pré-requis indispensable pour pouvoir utiliser ADB mais aussi pour pouvoir rooter l&aposappareil.

Purple Drake

Pour le rooting à proprement parlé, le sésame s&aposappelle Purple Drake. Téléchargez donc la dernière version (R3 dans mon cas) et décompressez là en local.

Étape spécial "Wheezy"

Pour ceux qui sont sur une version récente du kernel, cette étape peut être zappée. Dans mon cas, la version de GLibC incluse dans Debian Wheezy ne me permet de lancer le script tel quel avec les versions binaires d&aposadb incluse dans l&aposarchive.

Cela se règle cependant simplement en installant la version debian d&aposadb par sudo apt-get install android-adb. Ceci fait, allez dans le dossier assets de l&aposarchive décompressée de Purple Drake et éditez le fichier purpledrake_main.sh pour remplacer en ligne 16, $1 par /usr/bin.

Lorsque tout est en ordre, il ne reste plus qu&aposà lancer l&aposoutil. Assurez-vous que la tablette n&aposa aucune application de lancée (rebootez là si nécessaire), que le cable USB est bien connecté, et que dans la barre de status de la tablette vous voyez bien que la connection USB se fait en mode debuggage.

Si tout est OK, lancez le rooting par sudo ./purpledrake_linux.sh. La raison du sudo ici est que sous Debian, en standard, seul root a accès au device USB. Cela peut se configurer au niveau d&aposudev mais ça me saoule un peu d&aposavoir à faire cela à chaque fois, sur chaque machine et pour chaque device. Et c&aposest d&aposailleurs l&aposune des raisons qui me fait passer ADB en mode réseau.

Une fois l&aposoutil lancé il suffit de se laisser guider par le script. Il va d&aposabord rebooter le device, puis installer un root temporaire, puis l&aposutiliser, si vous le désirez, pour mettre en place le root permanent.

Une fois la tablette redémarrée, tout application qui demande l&aposaccès root devrait ainsi l&aposobtenir. C&aposest une première étape mais ce n&aposest pas très sécurisé, loin de là.

Super pouvoirs à la demande

Pour aller un cran plus loin, rendez vous sur le market et téléchargez SuperSU. Cette application au démarrage va remplacer la commande su fournie par Purple Drake, par une version qui vous demandera si telle application a bien le droit d&aposobtenir l&aposaccès root. Au premier lancement, SuperSU va vous proposer une procédure de remplacement de la commande en mode normal ou recovery. J&aposai personnellement pris l&aposoption normal. Et après une installation avec succès suivi d&aposun redémarrage de l&aposengin, tout était opérationnel.

Tester le tout

Pour tester, le plus simple est d&aposinstaller ADB Wifi à partir du market et de le lancer. Si les étapes précédentes ont fonctionnées, lorsque vous activerez la ADB en mode réseau, une boite de dialogue doit apparaître pour valider l&aposaccès.

Ceci fait, ADB Wifi doit vous indiquer qu&aposil est en écoute et vous fournis l&aposIP de connection. Vous n&aposavez alors plus qu&aposà vous connecter de votre machine comme ceci :

$sudo adb connect 192.168.154.21
connected to 192.168.154.21:5555
$sudo adb shell

Conclusion

Il est toujours navrant d&aposavoir à perdre du temps sur quelque chose d&aposaussi trivial. Espérons que peu à peu les constructeurs s&aposinspire de FairPhone et arrêtent ainsi de prendre leurs clients pour des crétins... On peut toujours rêver ;-)

```

jeudi 1 janvier 2015 à 13h21

mardi 30 décembre 2014

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 29 décembre 2014

L'appel à conférenciers pour la PGCon 2015 (16-20 juin) a été lancé. Dépôt des candidatures jusqu'au 19 janvier 2014. Détails : http://www.pgcon.org/2015/papers.php

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Heikki Linnakangas a poussé :

Andrew Dunstan a poussé :

Peter Eisentraut a poussé :

  • Change local_preload_libraries to PGC_USERSET. This allows it to be used with ALTER ROLE SET. Although the old setting of PGC_BACKEND prevented changes after session start, after discussion it was more useful to allow ALTER ROLE SET instead and just document that changes during a session have no effect. This is similar to how session_preload_libraries works already. An alternative would be to change things to allow PGC_BACKEND and PGC_SU_BACKEND settings to be changed by ALTER ROLE SET. But that might need further research (e.g., log_connections would probably not work). based on patch by Kyotaro Horiguchi http://git.postgresql.org/pg/commitdiff/584e35d17c00b69d598522d3cf321fbd9b7dbd03

Álvaro Herrera a poussé :

  • get_object_address: separate domain constraints from table constraints. Apart from enabling comments on domain constraints, this enables a future project to replicate object dropping to remote servers: with the current mechanism there's no way to distinguish between the two types of constraints, so there's no way to know what to drop. Also added support for the domain constraint comments in psql's \dd and pg_dump. Catalog version bumped due to the change in ObjectType enum. http://git.postgresql.org/pg/commitdiff/7eca575d1c28f6eee2bf4564f3d458d10c4a8f47
  • Use a bitmask to represent role attributes. The previous representation using a boolean column for each attribute would not scale as well as we want to add further attributes. Extra auxilliary functions are added to go along with this change, to make up for the lost convenience of access of the old representation. Catalog version bumped due to change in catalogs and the new functions. Author: Adam Brightwell, minor tweaks by Álvaro Reviewed by: Stephen Frost, Andres Freund, Álvaro Herrera http://git.postgresql.org/pg/commitdiff/1826987a46d079458007b7b6bbcbbd852353adbb
  • Add SQL-callable pg_get_object_address. This allows access to get_object_address from SQL, which is useful to obtain OID addressing information from data equivalent to that emitted by the parser. This is necessary infrastructure of a project to let replication systems propagate object dropping events to remote servers, where the schema might be different than the server originating the DROP. This patch also adds support for OBJECT_DEFAULT to get_object_address; that is, it is now possible to refer to a column's default value. Catalog version bumped due to the new function. Reviewed by Stephen Frost, Heikki Linnakangas, Robert Haas, Andres Freund, Abhijit Menon-Sen, Adam Brightwell. http://git.postgresql.org/pg/commitdiff/d7ee82e50f624221db76023c17137661fe69ec61
  • Revert "Use a bitmask to represent role attributes". This reverts commit 1826987a46d079458007b7b6bbcbbd852353adbb. The overall design was deemed unacceptable, in discussion following the previous commit message; we might find some parts of it still salvageable, but I don't want to be on the hook for fixing it, so let's wait until we have a new patch. http://git.postgresql.org/pg/commitdiff/a609d96778c1714b9af916477b2c30891fbe578a
  • Grab heavyweight tuple lock only before sleeping. We were trying to acquire the lock even when we were subsequently not sleeping in some other transaction, which opens us up unnecessarily to deadlocks. In particular, this is troublesome if an update tries to lock an updated version of a tuple and finds itself doing EvalPlanQual update chain walking; more than two sessions doing this concurrently will find themselves sleeping on each other because the HW tuple lock acquisition in heap_lock_tuple called from EvalPlanQualFetch races with the same tuple lock being acquired in heap_update -- one of these sessions sleeps on the other one to finish while holding the tuple lock, and the other one sleeps on the tuple lock. Per trouble report from Andrew Sackville-West in http://www.postgresql.org/message-id/20140731233051.GN17765@andrew-ThinkPad-X230 His scenario can be simplified down to a relatively simple isolationtester spec file which I don't include in this commit; the reason is that the current isolationtester is not able to deal with more than one blocked session concurrently and it blocks instead of raising the expected deadlock. In the future, if we improve isolationtester, it would be good to include the spec file in the isolation schedule. I posted it in http://www.postgresql.org/message-id/20141212205254.GC1768@alvh.no-ip.org Hat tip to Mark Kirkwood, who helped diagnose the trouble. http://git.postgresql.org/pg/commitdiff/0e5680f4737a9c6aa94aa9e77543e5de60411322
  • Remove event trigger from object_address test. It is causing trouble when run in parallel mode, because dropping the function other sessions are running concurrently causes them to fail due to inability to find the function. Per buildfarm, as noted by Tom Lane. http://git.postgresql.org/pg/commitdiff/289121a452e3c255e4fd63dbb0252261605c2eaf
  • Restrict name list len for domain constraints. This avoids an ugly-looking "cache lookup failure" message. Ugliness pointed out by Andres Freund. http://git.postgresql.org/pg/commitdiff/6630420fc9139c3a1ee051bc2f2ae8fa32e71c62

Tom Lane a poussé :

Bruce Momjian a poussé :

Fujii Masao a poussé :

Andres Freund a poussé :

  • Suppress MSVC warning in typeStringToTypeName function. MSVC doesn't realize ereport(ERROR) doesn't return. David Rowley http://git.postgresql.org/pg/commitdiff/cd5ebe1edd6d9c19caba406a90f626afec5d3f0a
  • Convert the PGPROC->lwWaitLink list into a dlist instead of open coding it. Besides being shorter and much easier to read it changes the logic in LWLockRelease() to release all shared lockers when waking up any. This can yield some significant performance improvements - and the fairness isn't really much worse than before, as we always allowed new shared lockers to jump the queue. http://git.postgresql.org/pg/commitdiff/7882c3b0b95640e361f1533fe0f2d02e4e5d8610
  • Add capability to suppress CONTEXT: messages to elog machinery. Hiding context messages usually is not a good idea - except for rather verbose debugging/development utensils like LOG_DEBUG. There the amount of repeated context messages just bloat the log without adding information. http://git.postgresql.org/pg/commitdiff/570bd2b3fd20d3f8896f5e6a8133d6aed385ac86
  • Lockless StrategyGetBuffer clock sweep hot path. StrategyGetBuffer() has proven to be a bottleneck in a number of buffer acquisition heavy workloads. To some degree this has already been alleviated by 5d7962c6, but it still can be quite a heavy bottleneck. The problem is that in unfortunate usage patterns a single StrategyGetBuffer() call will have to look at a large number of buffers - in turn making it likely that the process will be put to sleep while still holding the spinlock. Replace most of the usage of the buffer_strategy_lock spinlock for the clock sweep by a atomic nextVictimBuffer variable. That variable, modulo NBuffers, is the current hand of the clock sweep. The buffer clock-sweep then only needs to acquire the spinlock after a wraparound. And even then only in the process that did the wrapping around. That alleviates nearly all the contention on the relevant spinlock, although significant contention on the cacheline can still exist. Reviewed-By: Robert Haas and Amit Kapila. Discussion: 20141010160020.GG6670@alap3.anarazel.de, 20141027133218.GA2639@awork2.anarazel.de http://git.postgresql.org/pg/commitdiff/d72731a70450b5e7084991b9caa15cb58a2820df
  • Blindly fix a dtrace probe in lwlock.c for a removed local variable. Per buildfarm member locust. http://git.postgresql.org/pg/commitdiff/740a4ec7f4adccd56eced452580ccec7157fc8d8
  • Improve LWLock scalability. The old LWLock implementation had the problem that concurrent lock acquisitions required exclusively acquiring a spinlock. Often that could lead to acquirers waiting behind the spinlock, even if the actual LWLock was free. The new implementation doesn't acquire the spinlock when acquiring the lock itself. Instead the new atomic operations are used to atomically manipulate the state. Only the waitqueue, used solely in the slow path, is still protected by the spinlock. Check lwlock.c's header for an explanation about the used algorithm. For some common workloads on larger machines this can yield significant performance improvements. Particularly in read mostly workloads. Reviewed-By: Amit Kapila and Robert Haas. Author: Andres Freund. Discussion: 20130926225545.GB26663@awork2.anarazel.de http://git.postgresql.org/pg/commitdiff/ab5194e6f617a9a9e7aadb3dd1cee948a42d0755

Noah Misch a poussé :

  • Have config_sspi_auth() permit IPv6 localhost connections. Windows versions later than Windows Server 2003 map "localhost" to ::1. Account for that in the generated pg_hba.conf, fixing another oversight in commit f6dc6dd5ba54d52c0733aaafc50da2fbaeabb8b0. Back-patch to 9.0, like that commit. David Rowley and Noah Misch http://git.postgresql.org/pg/commitdiff/8d9cb0bc483431780aef483c7c110ab39e465958

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Michael Paquier sent in another revision of a patch to clarify some points about how numeric functions and operators work.
  • Etsuro Fujita sent in a patch to fix the fact that ExplainModifyTarget does not always show the parent of the inheritance tree in multi-target-table cases.
  • Heikki Linnakangas sent in another revision of a patch to advance local xmin more aggressively using the new pairing heap code.
  • Robert Haas sent in another revision of a patch to implement parallel mode and parallel contexts.
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to implement TABLESAMPLE.
  • Christoph Berg sent in a patch to correct the misapprehension that pg_upgrade requires "postmaster," an entity not present in all live versions of PostgreSQL, when "postgres" will do.
  • Michael Paquier sent in two more revisions of a patch to move contrib binaries to bin.
  • Álvaro Herrera sent in a flock of five patches intended to complete the feature which allows DROP to propagate via WAL.
  • Etsuro Fujita sent in two more revisions of a patch to allow foreign tables to be part of table inheritance hierarchies.
  • Heikki Linnakangas and Teodor Sigaev traded patches to support ranges with B-Tree GIN indexes.
  • Heikki Linnakangas sent in another revision of a patch to add a compress method for SP-GiST.
  • Andres Freund sent in another revision of a patch to add the capability to suppress CONTEXT: messages to elog machinery.
  • Andres Freund sent in another revision of a patch to convert the PGPROC->lwWaitLink list into a dlist instead of open coding it, and one to implement wait-free LW_SHARED LWLock acquisition.
  • Andres Freund sent in a patch to add a lockless StrategyGetBuffer clock sweep hotpath.
  • Michael Paquier sent in a patch to install libpq.dll in bin directory on Windows / Cygwin.
  • Andrew Dunstan sent in a patch to speed up the bin checks.
  • Teodor Sigaev sent in another revision of a patch to speed up tidbitmap using a cache page.
  • Jim Nasby sent in a patch to add a hack to test out several hashing strategies.
  • Andreas Karlsson sent in another revision of a patch to use 128-bit integers for sum, avg and statistics aggregates.
  • Bruce Momjian sent in a patch to fix an issue where misaligned BufferDescriptors is causing major performance problems on AMD.
  • Jeff Davis sent in another revision of a patch to implement spill-to-disk for HashAggs.
  • Alexander Shulgin sent in another revision of a patch to turn recovery.conf into GUCs.
  • Kaigai Kouhei sent in another revision of a patch to add a CTID scan as an example of the custom scan capability.
  • David Rowley and Noah Misch traded patches to update Windows with the changes that secured "make check."
  • Abhijit Menon-Sen sent in a patch to use CPUID at startup to determine whether the SSE4.2 CRC instructions are available, and to use them instead of the slice-by-8 implementation if they are.
  • Fujii Masao and Michael Paquier traded patches implementing compression of full-page writes.
  • Abhijit Menon-Sen sent in a patch to implement a fast bloat measurement tool.
  • Alexander Shulgin and Pavel Stehule traded patches to allow COPY to handle wrong rows without rolling back entirely.
  • Alexey Vasiliev sent in a patch to add recovery_timeout option to control timeout of restore_command nonzero status code.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT {UPDATE | IGNORE}.
  • Ian Lawrence Barwick sent in a patch to fix psql's tab completion of COMMENT ON not to repeat IS more than needed.
  • Amit Kapila and Dilip Kumar traded patches to enable vacuumdb to use parallel cores.

par N Bougain le mardi 30 décembre 2014 à 21h21

dimanche 28 décembre 2014

Guillaume Lelarge

Manuel de la 9.4 et articles GLMF

Trois billets en 2014 (avec celui-là), j'aurais fait fort :)

Néanmoins, quelques informations importantes.

Tout d'abord, la documentation de PostgreSQL 9.4 est enfin disponible entièrement en français, avec les différents formats habituels. C'est comme d'habitude sur http://docs.postgresql.fr.

Ensuite, au niveau des articles pour le GNU/Linux Magazine France, deux articles sont parus sur la version 9.4, un peu avant la sortie (les numéros 175 et 177). La suite de la série sur le planificateur est prévue rapidement avec un article sur les jointures. Il s'agit donc du deuxième article. Les troisième et quatrième articles sont déjà dans le bureau du rédacteur en chef (depuis août en fait). Ils devraient sortir rapidement après, suivant la place disponible dans le magazine.

Et voilà. L'année 2015 devrait être très intéressante :) Quelques infos plus tard à ce sujet.

par Guillaume Lelarge le dimanche 28 décembre 2014 à 11h36

mercredi 24 décembre 2014

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 21 décembre 2014

Publication de PostgreSQL 9.4 ! http://www.postgresql.org/docs/9.4/static/release-9-4.html
[fr : http://blog.postgresql.fr/index.php?post/2014/12/18/PostgreSQL-9.4.0]

Le groupe Meetup de Moscou célèbrera la sortie de PostgreSQL 9.4 le mardi 23 décembre dans les bureaux de Parallels. Détails et RSVP : http://PostgreSQLRussia.org

Le PUG de Bulgarie est sur pied : http://bgpug.org/

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Improve documentation around parameter-setting and ALTER SYSTEM. The ALTER SYSTEM ref page hadn't been held to a very high standard, nor was the feature well integrated into section 18.1 (parameter setting). Also, though commit 4c4654afe had improved the structure of 18.1, it also introduced a lot of poor wording, imprecision, and outright falsehoods. Try to clean that up. http://git.postgresql.org/pg/commitdiff/af06aa822dbc023f3a103278ec381e1c88d67db1
  • Fix point <-> polygon code for zero-distance case. "PG_RETURN_FLOAT8(x)" is not "return x", except perhaps by accident on some platforms. http://git.postgresql.org/pg/commitdiff/9418820efb32e739706cc3860347089315562ee4
  • Fix file descriptor leak after failure of a \setshell command in pgbench. If the called command fails to return data, runShellCommand forgot to pclose() the pipe before returning. This is fairly harmless in the current code, because pgbench would then abandon further processing of that client thread; so no more than nclients descriptors could be leaked this way. But it's not hard to imagine future improvements whereby that wouldn't be true. In any case, it's sloppy coding, so patch all branches. Found by Coverity. http://git.postgresql.org/pg/commitdiff/d38e8d30cecea96a088330133e232c465a222d0a
  • Suppress bogus statistics when pgbench failed to complete any transactions. Code added in 9.4 would attempt to divide by zero in such cases. Noted while testing fix for missing-pclose problem. http://git.postgresql.org/pg/commitdiff/de8e46f5f5785f4016aabf2aa231faa89a0746fb
  • Fix off-by-one loop count in MapArrayTypeName, and get rid of static array. MapArrayTypeName would copy up to NAMEDATALEN-1 bytes of the base type name, which of course is wrong: after prepending '_' there is only room for NAMEDATALEN-2 bytes. Aside from being the wrong result, this case would lead to overrunning the statically allocated work buffer. This would be a security bug if the function were ever used outside bootstrap mode, but it isn't, at least not in any currently supported branches. Aside from fixing the off-by-one loop logic, this patch gets rid of the static work buffer by having MapArrayTypeName pstrdup its result; the sole caller was already doing that, so this just requires moving the pstrdup call. This saves a few bytes but mainly it makes the API a lot cleaner. Back-patch on the off chance that there is some third-party code using MapArrayTypeName with less-secure input. Pushing pstrdup into the function should not cause any serious problems for such hypothetical code; at worst there might be a short term memory leak. Per Coverity scanning. http://git.postgresql.org/pg/commitdiff/66709133c7e5506be19bf56a82f45dd150f74732
  • Fix poorly worded error message. Adam Brightwell, per report from Martín Marqués. http://git.postgresql.org/pg/commitdiff/c977b8cffc76be00fbaab76e3271e05104260ec7
  • Fix another poorly worded error message. Spotted by Álvaro Herrera. http://git.postgresql.org/pg/commitdiff/c340494235111fb87e46b11ca6a87a7a43795f0f
  • Allow CHECK constraints to be placed on foreign tables. As with NOT NULL constraints, we consider that such constraints are merely reports of constraints that are being enforced by the remote server (or other underlying storage mechanism). Their only real use is to allow planner optimizations, for example in constraint-exclusion checks. Thus, the code changes here amount to little more than removal of the error that was formerly thrown for applying CHECK to a foreign table. (In passing, do a bit of cleanup of the ALTER FOREIGN TABLE reference page, which had accumulated some weird decisions about ordering etc.) Shigeru Hanada and Etsuro Fujita, reviewed by Kyotaro Horiguchi and Ashutosh Bapat. http://git.postgresql.org/pg/commitdiff/fc2ac1fb41c2defb8caf825781af75db158fb7a9
  • Improve hash_create's API for selecting simple-binary-key hash functions. Previously, if you wanted anything besides C-string hash keys, you had to specify a custom hashing function to hash_create(). Nearly all such callers were specifying tag_hash or oid_hash; which is tedious, and rather error-prone, since a caller could easily miss the opportunity to optimize by using hash_uint32 when appropriate. Replace this with a design whereby callers using simple binary-data keys just specify HASH_BLOBS and don't need to mess with specific support functions. hash_create() itself will take care of optimizing when the key size is four bytes. This nets out saving a few hundred bytes of code space, and offers a measurable performance improvement in tidbitmap.c (which was not exploiting the opportunity to use hash_uint32 for its 4-byte keys). There might be some wins elsewhere too, I didn't analyze closely. In future we could look into offering a similar optimized hashing function for 8-byte keys. Under this design that could be done in a centralized and machine-independent fashion, whereas getting it right for keys of platform-dependent sizes would've been notationally painful before. For the moment, the old way still works fine, so as not to break source code compatibility for loadable modules. Eventually we might want to remove tag_hash and friends from the exported API altogether, since there's no real need for them to be explicitly referenced from outside dynahash.c. Teodor Sigaev and Tom Lane http://git.postgresql.org/pg/commitdiff/4a14f13a0abfbf7e7d44a3d2689444d1806aa9dc
  • Improve documentation about CASE and constant subexpressions. The possibility that constant subexpressions of a CASE might be evaluated at planning time was touched on in 9.17.1 (CASE expressions), but it really ought to be explained in 4.2.14 (Expression Evaluation Rules) which is the primary discussion of such topics. Add text and an example there, and revise the <note> under CASE to link there. Back-patch to all supported branches, since it's acted like this for a long time (though 9.2+ is probably worse because of its more aggressive use of constant-folding via replanning of nominally-prepared statements). Pre-9.4, also back-patch text added in commit 0ce627d4 about CASE versus aggregate functions. Tom Lane and David Johnston, per discussion of bug #12273. http://git.postgresql.org/pg/commitdiff/5b516835897f2a487eedacdb605ac40d52f6b484
  • Docs: clarify treatment of variadic functions with zero variadic arguments. Explain that you have to use "VARIADIC ARRAY[]" to pass an empty array to a variadic parameter position. This was already implicit in the text but it seems better to spell it out. Per a suggestion from David Johnston, though I didn't use his proposed wording. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/699300a146c04e207a8fdec407538cdf5368fde5

Peter Eisentraut a poussé :

Heikki Linnakangas a poussé :

Álvaro Herrera a poussé :

Noah Misch a poussé :

Andrew Dunstan a poussé :

  • Fix some jsonb issues found by Coverity in recent commits. Mostly these issues concern the non-use of function results. These have been changed to use (void) pushJsonbValue(...) instead of assigning the result to a variable that gets overwritten before it is used. There is a larger issue that we should possibly examine the API for pushJsonbValue(), so that instead of returning a value it modifies a state argument. The current idiom is rather clumsy. However, changing that requires quite a bit more work, so this change should do for the moment. http://git.postgresql.org/pg/commitdiff/c8315930e6a34b616d9840985c85eb0d856dd2df

Magnus Hagander a poussé :

Andres Freund a poussé :

  • Fix (re-)starting from a basebackup taken off a standby after a failure. When starting up from a basebackup taken off a standby extra logic has to be applied to compute the point where the data directory is consistent. Normal base backups use a WAL record for that purpose, but that isn't possible on a standby. That logic had a error check ensuring that the cluster's control file indicates being in recovery. Unfortunately that check was too strict, disregarding the fact that the control file could also indicate that the cluster was shut down while in recovery. That's possible when the a cluster starting from a basebackup is shut down before the backup label has been removed. When everything goes well that's a short window, but when either restore_command or primary_conninfo isn't configured correctly the window can get much wider. That's because inbetween reading and unlinking the label we restore the last checkpoint from WAL which can need additional WAL. To fix simply also allow starting when the control file indicates "shutdown in recovery". There's nicer fixes imaginable, but they'd be more invasive. Backpatch to 9.2 where support for taking basebackups from standbys was added. http://git.postgresql.org/pg/commitdiff/c303e9e7e5d4ddf04526ed8e821ef4b1c3bc2d12
  • Adjust valgrind suppression to the changes in 2c03216d8311. CRC computation is now done in XLogRecordAssemble. http://git.postgresql.org/pg/commitdiff/72950dc1d02c6b44c6475485fd5ef8ac277bbdd0
  • Define Assert() et al to ((void)0) to avoid pedantic warnings. gcc's -Wempty-body warns about the current usage when compiling postgres without --enable-cassert. http://git.postgresql.org/pg/commitdiff/9959abb0122ca2b0e4817e20954e3083c90becdc
  • Prevent potentially hazardous compiler/cpu reordering during lwlock release. In LWLockRelease() (and in 9.4+ LWLockUpdateVar()) we release enqueued waiters using PGSemaphoreUnlock(). As there are other sources of such unlocks backends only wake up if MyProc->lwWaiting is set to false; which is only done in the aforementioned functions. Before this commit there were dangers because the store to lwWaitLink could become visible before the store to lwWaitLink. This could both happen due to compiler reordering (on most compilers) and on some platforms due to the CPU reordering stores. The possible consequence of this is that a backend stops waiting before lwWaitLink is set to NULL. If that backend then tries to acquire another lock and has to wait there the list could become corrupted once the lwWaitLink store is finally performed. Add a write memory barrier to prevent that issue. Unfortunately the barrier support has been only added in 9.2. Given that the issue has not knowingly been observed in praxis it seems sufficient to prohibit compiler reordering using volatile for 9.0 and 9.1. Actual problems due to compiler reordering are more likely anyway. Discussion: 20140210134625.GA15246@awork2.anarazel.de http://git.postgresql.org/pg/commitdiff/37de8de9e33606a043e98fee64b5595aedaa8254

Fujii Masao a poussé :

  • Remove odd blank line in comment. Etsuro Fujita http://git.postgresql.org/pg/commitdiff/26674c923d0ecbac2cda039433163d17584fae65
  • Update .gitignore for config.cache. Also add a comment about why regreesion.* aren't listed in .gitignore. Jim Nasby http://git.postgresql.org/pg/commitdiff/ccf292cd2ec16c69ddfee3bf72afe113a7595e00
  • Ensure variables live across calls in generate_series(numeric, numeric). In generate_series_step_numeric(), the variables "start_num" and "stop_num" may be potentially freed until the next call. So they should be put in the location which can survive across calls. But previously they were not, and which could cause incorrect behavior of generate_series(numeric, numeric). This commit fixes this problem by copying them on multi_call_memory_ctx. Andrew Gierth http://git.postgresql.org/pg/commitdiff/19e065c0492c34fbccbd2c3707ba68cff14195a3
  • Add memory barriers for PgBackendStatus.st_changecount protocol. st_changecount protocol needs the memory barriers to ensure that the apparent order of execution is as it desires. Otherwise, for example, the CPU might rearrange the code so that st_changecount is incremented twice before the modification on a machine with weak memory ordering. This surprising result can lead to bugs. This commit introduces the macros to load and store st_changecount with the memory barriers. These are called before and after PgBackendStatus entries are modified or copied into private memory, in order to prevent CPU from reordering PgBackendStatus access. Per discussion on pgsql-hackers, we decided not to back-patch this to 9.4 or before until we get an actual bug report about this. Patch by me. Review by Robert Haas. http://git.postgresql.org/pg/commitdiff/38628db8d8caff21eb6cf8d775c0b2d04cf07b9b

Bruce Momjian a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Peter Eisentraut sent in another revision of a patch to implement TRANSFORMs.
  • Kaigai Kouhei sent in a patch to add ctidscan as an example of the custom scan interface.
  • Amit Langote sent in a patch to fix a comment typo in typedef struct BrinTuple.
  • Marko (johto) Tiikkaja sent in a patch to add a DROP PRIVILEGES OWNED BY functionality.
  • Peter Eisentraut and Michael Paquier traded patches to move the executables from contrib/ to bin/
  • Petr (PJMODOS) Jelinek sent in a patch to add a few helper functions for making logical replication easier.
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to add a sequence access method.
  • Alexander Shulgin sent in another revision of a patch to add an ssl_protocols configuration option.
  • Shigeru HANADA sent in a patch to add push-down JOIN support for foreign tables.
  • Simon Riggs sent in two revisions of a patch to ensure that logical decoding follows timelines.
  • Michael Paquier sent in a patch to document some modes of vcregress.pl that had not previously been documented.
  • Michael Paquier sent in a patch to ensure that analyze_new_cluster.bat and delete_old_cluster.bat not be ignored with vcregress upgradecheck.
  • Kyotaro HORIGUCHI sent in another revision of a patch to enable asynchronous execution in postgres_fdw.
  • Simon Riggs sent in a patch to start WALWriter during recovery and make it responsible for fsyncing data, allowing WALReceiver to progress other useful actions.
  • Christoph Berg sent in a patch to fix an unlikely integer overflow.
  • Dilip Kumar sent in another revision of a patch to allow parallel cores to be used by vacuumdb.
  • Amit Kapila sent in two more revisions of a patch to fix a mismatch between pg_basebackup and Windows.
  • Alexander Shulgin sent in a patch to allow a different HINT to be sent to the server error log and to the client, which will be useful where there's security sensitive information that's more appropriate for a HINT than a DETAIL message. Alongside this, he sent a patch to log a hint if pg_ident.conf or pg_hba.conf changed since last reload.
  • Heikki Linnakangas sent in four more revisions of a patch to implement a GiST kNN search queue.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT {UPDATE | IGNORE}, and a patch on top of same to fix a bug.
  • Adam Brightwell and Álvaro Herrera traded patches to implement a role attribute bitmask catalog representation.
  • Tomas Vondra and Ali Akbar traded patches to decrease memory needlessly consumed by array_agg.
  • Teodor Sigaev sent in another revision of a patch to add a compress method for SP-GiST.
  • Marius Timmer sent in another revision of a patch to display sort order options in VERBOSE mode of EXPLAIN.
  • Alexander Shulgin sent in two more revisions of a patch to track TRUNCATE via pgstat.
  • Teodor Sigaev and David Rowley traded patches to speed up tidmap by caching pages.
  • Michael Paquier sent in four more revisions of a patch to compress full-page writes.
  • Teodor Sigaev sent in two more revisions of a patch to speed up tidmap by hashing BlockNumber.
  • Simon Riggs sent in another revision of a patch to turn off HOT/Cleanup sometimes.
  • Simon Riggs and David Rowley traded patches intended to make it possible to treat aggregates more flexibly, including strategies for parallelizing.
  • Tom Lane sent in a patch to make dynahash select normal hash functions.
  • Heikki Linnakangas sent in a patch to turn many of the btree_gin macros into real functions.
  • Mark Dilger sent in another revision of a patch to allow Oid formatting in printf/elog strings.
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to implement TABLESAMPLE.
  • Amit Kapila sent in another revision of a patch to implement parallel sequential scan.
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to keep track of transaction commit timestamps.
  • Heikki Linnakangas sent in a patch to fix an issue with bogus WAL segments archived after promotion by removing the suspect segments at timeline switch.
  • Heikki Linnakangas sent in a patch to make WAL archival behave more sensibly in standby mode.
  • Álvaro Herrera sent in a patch to distinguish domain constraint from table constraints.
  • Robert Haas sent in another revision of a patch to implement parallel mode and parallel contexts.
  • Peter Geoghegan sent in a patch to add a new HINT -- a guess as to what column the user might have intended to reference, to be shown in various contexts where an ERRCODE_UNDEFINED_COLUMN error is raised.
  • Andrew Dunstan sent in another revision of a patch to fix psql's over-eager use of pagers via a pager_min_lines setting.
  • Tatsuo Ishii sent in another revision of a patch to fix an infelicity between pgbench -f and vacuum.
  • Michael Paquier sent in a patch to fix some make problems with the addition of test programs to contrib.

par N Bougain le mercredi 24 décembre 2014 à 00h31

jeudi 18 décembre 2014

Actualités PostgreSQL.fr

PostgreSQL 9.4.0


Contenu

  Texte original
  Détail des fonctionnalités
  Téléchargement
  Documentation
  Licence
  Contacts
  Images et Logos
  Informations concernant les sociétés citées
  Support professionnel

Texte original

PostgreSQL 9.4 améliore la flexibilité, l'évolutivité et les performances

18 DÉCEMBRE 2014 : Le PostgreSQL Global Development Group annonce la sortie de PostgreSQL 9.4. Il s'agit de la dernière version en date du système de gestion de bases de données libre de référence. Cette version ajoute de nombreuses fonctionnalités qui améliorent la flexibilité, l'évolutivité et les performances de PostgreSQL. Et cela pour de nombreux cas d'utilisation différents, avec notamment l'amélioration du support de JSON, de la réplication et de la performance de l'indexation.

Flexibilité

Avec le nouveau type de données JSONB, plus besoin de choisir entre le stockage relationnel et non-relationnel : il y a les deux à la fois. JSONB supporte les recherches rapides et les requêtes de recherche d'expressions simples en utilisant les index de type Generalized Inverted Indexes (GIN). De nombreuses nouvelles fonctions permettent de manipuler les données JSON, avec des performances souvent meilleures que celles obtenues avec les bases de données orientées document les plus populaires. Avec JSONB, les données des tables peuvent être facilement intégrées à celles des documents permettant d'obtenir un environnement de bases de données complètement intégré.

« JSONB rapproche PostgreSQL des développeurs de la communauté JavaScript en permettant aux données JSON d'être stockées et requêtées nativement. node.js et les autres frameworks JavaScript côté serveur peuvent désormais profiter de la sécurité et de la robustesse de PostgreSQL, tout en continuant à stocker les données dans le format sans schéma qu'ils préfèrent », précise Matt Soldo, Responsable Produit chez Heroku Postgres.

Evolutivité

Avec la 9.4, le décodage logique (Logical Decoding) offre une nouvelle API pour lire, filtrer et manipuler le flux de réplication de PostgreSQL. Cette interface est la fondation de nouveaux outils de réplication, comme la réplication bi-directionnelle (Bi-Directional Replication), qui permet la création de grappes PostgreSQL multi-maîtres. D'autres améliorations dans la réplication système concernent l'administration et l'utilisation des réplicats, notamment les connecteurs de réplication et les réplicats différés.

« La raison principale à notre adoption immédiate de PostgreSQL 9.4 en production est la nouvelle fonctionnalité de décodage logique », explique Marco Favale, responsable de production Cartographique chez Navionics. « La possibilité d'écrire des greffons de sortie personnalisés et flexibles nous permettra de récupérer de manière transparente les modifications sur certaines tables et de les répliquer où nous voulons, tout en supprimant les contraintes relatives aux solutions de réplication par triggers, lourdes et plus complexes à gérer. »

« Zalando dépend de la stabilité et des performances de centaines de serveurs de bases de données PostgreSQL pour satisfaire des millions de clients partout en Europe », annonce Valentine Gogichashvili, Team Lead Database Operations chez Zalando Technologies. « Nous sommes impatients d'utiliser la réplication différée, immédiatement utilisable, et nous allons évaluer les outils de réplication bi-directionnelle dès qu'ils seront disponibles. »

Performances

La version 9.4 apporte également de multiples gains de performance, permettant aux utilisateurs de tirer plus d'avantages de leur serveur PostgreSQL dont :

  • des améliorations des index GIN, avec une taille réduite de 50% et une vitesse multipliée par 3 ;
  • des vues matérialisées qui peuvent être mises à jour de manière concurrente, pour du reporting plus rapide et plus à jour ;
  • le rechargement rapide du cache de base de données au redémarrage avec pg_prewarm ;
  • une écriture parallèle plus rapide des journaux de transaction de PostgreSQL.

« Nous allons vraiment beaucoup gagner avec le rafraichissement concurrent des vues matérialisées, les réplicats différés (qui permettront aux restaurations après incident d'être plus robustes), ainsi qu'avec les améliorations de performances apportées par chaque nouvelle version », ajoute Marco Favale.

Détail des fonctionnalités

Plus d'informations sur les fonctionnalités ci-dessus et les autres dans les liens suivants :

Téléchargement

Documentation

La documentation au format HTML et les pages de manuel sont installées avec PostgreSQL. La documentation en ligne, exhaustive et interactive, peut être parcourue, interrogée et commentée librement.

Licence

PostgreSQL utilise la licence PostgreSQL, une licence permissive de type BSD. Cette licence certifiée par l'OSI est largement appréciée pour sa flexibilité et sa compatibilité avec le monde des affaires, puisqu'elle ne restreint pas l'utilisation de PostgreSQL dans les applications propriétaires ou commerciales. Associée à un support proposé par de multiples sociétés et une propriété publique du code, sa licence rend PostgreSQL très populaire parmi les revendeurs souhaitant embarquer une base de données dans leurs produits sans avoir à se soucier des prix de licence, des verrous commerciaux ou modifications des termes de licence.

Contacts

Pages Web

Contact presse

France et pays francophones
Stéphane Schildknecht
fr at postgresql dot org
+33 (0) 617 11 37 42

Pour les contacts d'autres régions, consulter la liste des contacts internationaux.

Images et Logos

Tous les logos sont modifiables et redistribuables selon les termes de la licence PostgreSQL. Le nom PostgreSQL et le logo sont des marques déposées de la PostgreSQL Community Association of Canada.

Informations concernant les sociétés citées

« JSONB rapproche PostgreSQL des développeurs de la communauté JavaScript en permettant aux données JSON d'être stockées et requêtées nativement. node.js et les autres frameworks JavaScript côté serveur peuvent désormais profiter de la sécurité et de la robustesse de PostgreSQL, tout en continuant à stocker les données dans le format sans schéma qu'ils préfèrent », précise Matt Soldo, Responsable Produit chez Heroku Postgres.

À propos de Heroku : Heroku est le leader des plateformes en tant que service (PaaS), orienté sur la facilité d'utilisation, l'automatisation, et la fiabilité. Heroku, dont le siège se trouve à San Francisco, en Californie, est un supporter passioné et actif sur les technologies Ruby et les architectures en nuage. Pour plus d'informations, visitez notre site web et notre blog ou suivez Heroku sur Twitter. Heroku est une filiale, propriété exclusive de salesforce.com. Contact (uniquement en Anglais): Heroku PR

« La raison principale à notre adoption immédiate de PostgreSQL 9.4 en production est la nouvelle fonctionnalité de décodage logique », explique Marco Favale, responsable de production Cartographique chez Navionics. « La possibilité d'écrire des greffons de sortie personnalisés et flexibles nous permettra de récupérer de manière transparente les modifications sur certaines tables et de les répliquer où nous voulons, tout en supprimant les contraintes relatives aux solutions de réplication par triggers, lourdes et plus complexes à gérer. »

« Nous allons vraiment beaucoup gagner avec le rafraichissement concurrent des vues matérialisées, les réplicats différés (qui permettront aux restaurations après incident d'être plus robustes), ainsi qu'avec les améliorations de performances apportées par chaque nouvelle version », ajoute Marco Favale.

À propos de Navionics : Navionics (www.navionics.com) dispose de la plus important base de données de cartes marines, de voies navigables et de lacs au monde, couvrant les océans, toutes les mers de la planète ainsi que des centaines de milliers de lacs et rivières. Nombre de ces cartes ont été élaborées par le biais d'enquêtes exclusives de Navionics réalisées à la fois sur le terrain et par sondage à distance notamment par imagerie satellitaire et scanners laser aéroportés. Navionics est situé en Italie, aux États-Unis, en Inde et en Estonie, avec des ventes et des services partout dans le monde. Contact (Anglais ou Italien): Marco Favale

« Zalando dépend de la stabilité et des performances de centaines de serveurs de bases de données PostgreSQL pour satisfaire des millions de clients partout en Europe. Nous nous tenons à jour des dernières versions de développement de PostgreSQL depuis 2010, lorsque nous avons mis en place la première Release Candidate de PostgreSQL 9.0. Chaque nouvelle version nous apporte plus de performances et des améliorations dont nous bénéficions pratiquement dès le premier jour. Nous sommes impatients d'utiliser la réplication différée, immédiatement utilisable avec PostgreSQL 9.4. Nous allons évaluer les outils de réplication bi-directionnelle, rendus possibles avec l'introduction du décodage logique. Notre équipe travaille déjà à l'amélioration de nos outils de supervision open source qui bénéficieront des statistiques temps réel sur les commandes fournies par le module pg_stat_statements et les multiples améliorations qu'il a reçu dans la version 9.4. Les améliorations des aggrégats, comme filter, rendent l'écriture des requêtes avec aggrégats plus facile et encourangent par la suite nos collègues à écrire du SQL bien plus élégant qu'auparavant », explique Valentine Gogichashvili, responsable des opérations bases de données chez Zalando Technologies.

Zalando est l'une des plus grosses entreprises de e-commerce en europe, opérant dans 15 pays à ce jour. Chez Zalando Technology, nous avons développé l'essentiel de notre plateforme en interne, notamment la boutique, les systèmes internes et le logiciel de logistique. En ce qui concerne le développement et les opérations sur les systèmes, nous utilisons majoritairement des solutions open source et travaillons en petites équipes flexibles. Nos équipes technologiques créent l'incroyable aventure de shopping en ligne Zalando, que des millions de clients apprécient chaque jour. Venez visiter notre site web ou notre blog

Support professionnel

PostgreSQL bénéficie du support de nombreuses sociétés, qui financent des développeurs, fournissent l'hébergement ou un support financier. Les plus fervents supporters sont listés sur la page des mécènes du développement.

Le travail sur JSONB a été financé par Engine Yard et soutenu par Heroku, Andrew Dunstan et plusieurs autres contributeurs. Le travail sur le Décodeur Logique a été piloté par 2ndQuadrant. Plusieurs autres sociétés ont contribué à cette version.

Il existe également une très grande communauté de sociétés fournissant du support PostgreSQL, des consultants indépendants aux sociétés multinationales.

Les dons sont acceptés avec plaisir.

Vous pouvez également acheter des produits dérivés de qualité sur la Boutique Zazzle PostgreSQL.

par SAS le jeudi 18 décembre 2014 à 17h57

mardi 16 décembre 2014

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 14 décembre 2014

Le meetup du 18 février 2015 à Melbourne accueillera Gabriele Bartolini à propos de PostgreSQL 9.4 pour les devops. Détails ci-après, et RSVP : http://www.meetup.com/melpug/events/219082475/

Le Prague PostgreSQL Developer Day (P2D2) 2015 aura lieu à Prague, République Tchèque, les 11 & 12 février 2015 : http://www.p2d2.cz/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Simon Riggs a poussé :

Peter Eisentraut a poussé :

Álvaro Herrera a poussé :

Heikki Linnakangas a poussé :

Tom Lane a poussé :

  • Fix minor thinko in convertToJsonb(). The amount of space to reserve for the value's varlena header is VARHDRSZ, not sizeof(VARHDRSZ). The latter coding accidentally failed to fail because of the way the VARHDRSZ macro is currently defined; but if we ever change it to return size_t (as one might reasonably expect it to do), convertToJsonb() would have failed. Spotted by Mark Dilger. http://git.postgresql.org/pg/commitdiff/24688f4e5a7d5fadde0e43a5b123432d81577d82
  • Fix assorted confusion between Oid and int32. In passing, also make some debugging elog's in pgstat.c a bit more consistently worded. Back-patch as far as applicable (9.3 or 9.4; none of these mistakes are really old). Mark Dilger identified and patched the type violations; the message rewordings are mine. http://git.postgresql.org/pg/commitdiff/06d5803ffa1ea5a42295c9266f30557e44a99de7
  • Fix completely broken REINDEX SCHEMA testcase. Aside from not testing the case it claimed to test (namely a permissions failure), it left a login-capable role lying around, which quite aside from possibly being a security hole would cause subsequent regression runs to fail since the role would already exist. http://git.postgresql.org/pg/commitdiff/58af84f4bba5c87b352cf40e276e287fd289bd77
  • Make rowsecurity test clean up after itself, too. Leaving global objects like roles hanging around is bad practice. http://git.postgresql.org/pg/commitdiff/0845264642d855d92c63c5d05a4ef83245ca16c5
  • Fix corner case where SELECT FOR UPDATE could return a row twice. In READ COMMITTED mode, if a SELECT FOR UPDATE discovers it has to redo WHERE-clause checking on rows that have been updated since the SELECT's snapshot, it invokes EvalPlanQual processing to do that. If this first occurs within a non-first child table of an inheritance tree, the previous coding could accidentally re-return a matching row from an earlier, already-scanned child table. (And, to add insult to injury, I think this could make it miss returning a row that should have been returned, if the updated row that this happens on should still have passed the WHERE qual.) Per report from Kyotaro Horiguchi; the added isolation test is based on his test case. This has been broken for quite awhile, so back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/2db576ba8c449fcaf61ae7aa14ed62e63ebf5924
  • Fix planning of SELECT FOR UPDATE on child table with partial index. Ordinarily we can omit checking of a WHERE condition that matches a partial index's condition, when we are using an indexscan on that partial index. However, in SELECT FOR UPDATE we must include the "redundant" filter condition in the plan so that it gets checked properly in an EvalPlanQual recheck. The planner got this mostly right, but improperly omitted the filter condition if the index in question was on an inheritance child table. In READ COMMITTED mode, this could result in incorrectly returning just-updated rows that no longer satisfy the filter condition. The cause of the error is using get_parse_rowmark() when get_plan_rowmark() is what should be used during planning. In 9.3 and up, also fix the same mistake in contrib/postgres_fdw. It's currently harmless there (for lack of inheritance support) but wrong is wrong, and the incorrect code might get copied to someplace where it's more significant. Report and fix by Kyotaro Horiguchi. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/462bd95705a0c23ba0b0ba60a78d32566a0384c1
  • Revert misguided change to postgres_fdw FOR UPDATE/SHARE code. In commit 462bd95705a0c23ba0b0ba60a78d32566a0384c1, I changed postgres_fdw to rely on get_plan_rowmark() instead of get_parse_rowmark(). I still think that's a good idea in the long run, but as Etsuro Fujita pointed out, it doesn't work today because planner.c forces PlanRowMarks to have markType = ROW_MARK_COPY for all foreign tables. There's no urgent reason to change this in the back branches, so let's just revert that part of yesterday's commit rather than trying to design a better solution under time pressure. Also, add a regression test case showing what postgres_fdw does with FOR UPDATE/SHARE. I'd blithely assumed there was one already, else I'd have realized yesterday that this code didn't work. http://git.postgresql.org/pg/commitdiff/8ec8760fc87ecde0516e511f1c55aec627b01ea7
  • Avoid instability in output of new REINDEX SCHEMA test. The planner seems to like to do this join query as a hash join, making the output ordering machine-dependent; worse, it's a hash on OIDs, so that it's a bit astonishing that the result doesn't change from run to run even on one machine. Add an ORDER BY to get consistent results. Per buildfarm. I also suppressed output from the final DROP SCHEMA CASCADE, to avoid occasional failures similar to those fixed in commit 81d815dc3ed74a7d. That hasn't been observed in the buildfarm yet, but it seems likely to happen in future if we leave it as-is. http://git.postgresql.org/pg/commitdiff/1c5c70df45714f38c033bb1a272d8db4f2dc8a2f
  • Repair corner-case bug in array version of percentile_cont(). The code for advancing through the input rows overlooked the case that we might already be past the first row of the row pair now being considered, in case the previous percentile also fell between the same two input rows. Report and patch by Andrew Gierth; logic rewritten a bit for clarity by me. http://git.postgresql.org/pg/commitdiff/b0f479113abaf28f1c9ed25d856ffe6fd4bb7dc9
  • Improve recovery target settings documentation. Commit 815d71dee hadn't bothered to update the documentation to match the behavioral change, and a lot of other text in this section was badly in need of copy-editing. http://git.postgresql.org/pg/commitdiff/e311cd6ded096122a5f2b5cbe91bc3a9f0dda3cb
  • Update 9.4 release notes. Set release date, do a final pass of wordsmithing, improve some other new-in-9.4 documentation. http://git.postgresql.org/pg/commitdiff/0923b01e3e64d4985ffbdf8ec912e25ad02e0c0c

Andrew Dunstan a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Petr (PJMODOS) Jelinek sent in two more revisions of a patch to add a shutdown_at_recovery_target option to recovery.conf.
  • Euler Taveira de Oliveira sent in a patch to bring the documentation of pg_receivexlog into line with its new functionality.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT {UPDATE | IGNORE}.
  • Petr (PJMODOS) Jelinek sent in another revision of a patch to add a sequence access method.
  • Alexander Shulgin sent in two revisions of a patch to track TRUNCATEs in pgstat transaction stats.
  • Robert Haas and Heikki Linnakangas traded patches to advance local xmin more aggressively.
  • Etsuro Fujita sent in another revision of a patch to allow foreign tables to be part of table inheritance hierarchies.
  • Álvaro Herrera sent in a patch to implement logical column ordering.
  • David Rowley sent in another revision of a patch to allow removing INNER JOIN under some circumstances.
  • Matt Newell sent in another revision of a patch to enable libpq pipelining.
  • Heikki Linnakangas improved the performance of k-Nearest-Neighbor GiST index searches using a Pairing Heap http://en.wikipedia.org/wiki/Pairing_heap.
  • Petr (PJMODOS) Jelinek sent in a patch to implement TABLESAMPLE.
  • Jeff Davis sent in another revision of a patch to implement a memory-bounded HashAgg.
  • Mark Dilger sent in a WIP patch to allow printf formatting for Oids.
  • Kyotaro HORIGUCHI and Tom Lane traded patches to fix an issue where some queries return duplicate rows after FOR UPDATE was blocked, in other words, after getting HeapTupleUpdated in ExecLockRows.
  • Simon Riggs sent in another revision of a patch to turn off heap-only tuple cleanup under some circumstances.
  • Heikki Linnakangas and Michael Paquier traded patches to refactor the code for sync node detection.
  • Peter Eisentraut sent in a patch to ensure that VPATH builds not write to the source tree.
  • Kyotaro HORIGUCHI sent in a patch to fix an issue where SELECT FROM <inheritance parent> WHERE <cond> FOR UPDATE may return results which does not match the <cond>.
  • Alexander Shulgin sent in another revision of a patch to turn recovery.conf into GUCs.
  • Robert Haas sent in a patch to add two new concepts: parallel mode, and parallel contexts.
  • Heikki Linnakangas sent in a patch to place pg_rewind, a tool that allows repurposing an old master server as a new standby server, after promotion, even if the old master was not shut down cleanly, in contrib/.
  • Andreas Karlsson sent in a patch to reduce the required lock strength of trigger and foreign key DDL.
  • Tatsuo Ishii sent in a patch to pgbench from exiting when the pg_bench_* tables do not exist.
  • Michael Paquier sent in another revision of a patch to allow compressing full-page writes.
  • Ali Akbar sent in a test for the patch to fix xpath() to return namespace definitions.
  • Emre Hasegeli sent in another revision of a patch to implement a BRIN range operator class.

par N Bougain le mardi 16 décembre 2014 à 23h49

lundi 8 décembre 2014

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 7 décembre 2014

Le Prague PostgreSQL Developer Day (P2D2) 2015 aura lieu à Prague, République Tchèque, les 11 & 12 février 2015 : http://www.p2d2.cz/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Tom Lane a poussé :

  • Add a #define for the inet overlaps operator. Extracted from pending inet selectivity patch. The rest of it isn't quite ready to commit, but we might as well push this part so the patch doesn't have to track the moving target of pg_operator.h. http://git.postgresql.org/pg/commitdiff/866737c923730f0cf8dde787868ab3ae697a278a
  • Guard against bad "dscale" values in numeric_recv(). We were not checking to see if the supplied dscale was valid for the given digit array when receiving binary-format numeric values. While dscale can validly be more than the number of nonzero fractional digits, it shouldn't be less; that case causes fractional digits to be hidden on display even though they're there and participate in arithmetic. Bug #12053 from Tommaso Sala indicates that there's at least one broken client library out there that sometimes supplies an incorrect dscale value, leading to strange behavior. This suggests that simply throwing an error might not be the best response; it would lead to failures in applications that might seem to be working fine today. What seems the least risky fix is to truncate away any digits that would be hidden by dscale. This preserves the existing behavior in terms of what will be printed for the transmitted value, while preventing subsequent arithmetic from producing results inconsistent with that. In passing, throw a specific error for the case of dscale being outside the range that will fit into a numeric's header. Before you got "value overflows numeric format", which is a bit misleading. Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/0927bf8060eca4e220cde48b1425e7bcf7451b94
  • Minor cleanup of function declarations for BRIN. Get rid of PG_FUNCTION_INFO_V1() macros, which are quite inappropriate for built-in functions (possibly leftovers from testing as a loadable module?). Also, fix gratuitous inconsistency between SQL-level and C-level names of the minmax support functions. http://git.postgresql.org/pg/commitdiff/1511521a36f99a242f51c3bd0811cfcd53c5e63a
  • Fix JSON aggregates to work properly when final function is re-executed. Davide S. reported that json_agg() sometimes produced multiple trailing right brackets. This turns out to be because json_agg_finalfn() attaches the final right bracket, and was doing so by modifying the aggregate state in-place. That's verboten, though unfortunately it seems there's no way for nodeAgg.c to check for such mistakes. Fix that back to 9.3 where the broken code was introduced. In 9.4 and HEAD, likewise fix json_object_agg(), which had copied the erroneous logic. Make some cosmetic cleanups as well. http://git.postgresql.org/pg/commitdiff/75ef435218fe4f6ce29d33e10f6f7a328c93f7e7
  • Improve error messages for malformed array input strings. Make the error messages issued by array_in() uniformly follow the style ERROR: malformed array literal: "actual input string" DETAIL: specific complaint here and rewrite many of the specific complaints to be clearer. The immediate motivation for doing this is a complaint from Josh Berkus that json_to_record() produced an unintelligible error message when dealing with an array item, because it tries to feed the JSON-format array value to array_in(). Really it ought to be smart enough to perform JSON-to-Postgres array conversion, but that's a future feature not a bug fix. In the meantime, this change is something we agreed we could back-patch into 9.4, and it should help de-confuse things a bit. http://git.postgresql.org/pg/commitdiff/475aedd1ef0c0f9fc9d675dd2286380d14804975

Magnus Hagander a poussé :

Heikki Linnakangas a poussé :

Andrew Dunstan a poussé :

  • Fix hstore_to_json_loose's detection of valid JSON number values. We expose a function IsValidJsonNumber that internally calls the lexer for json numbers. That allows us to use the same test everywhere, instead of inventing a broken test for hstore conversions. The new function is also used in datum_to_json, replacing the code that is now moved to the new function. Backpatch to 9.3 where hstore_to_json_loose was introduced. http://git.postgresql.org/pg/commitdiff/e09996ff8dee3f70b0a027cffebccef4388ed5b7

Álvaro Herrera a poussé :

  • dummy_seclabel: add sql/, expected/, and .gitignores. Michael Paquier http://git.postgresql.org/pg/commitdiff/332562437724845635f0e11685d11fdfe8719eb2
  • pageinspect/BRIN: minor tweaks. Michael Paquier. Double-dash additions suggested by Peter Geoghegan http://git.postgresql.org/pg/commitdiff/b52cb4690e0752efea440173c4923d76d2126679
  • Install kludges to fix check-world for src/test/modules. check-world failed in a completely clean tree, because src/test/modules fail to build unless errcodes.h is generated first. To fix this, install a dependency in src/test/modules' Makefile so that the necessary file is generated. Even with this, running "make check" within individual module subdirs will still fail because the dependency is not considered there, but this case is less interesting and would be messier to fix. check-world still failed with the above fix in place, this time because dummy_seclabel used LOAD to load the dynamic library, which doesn't work because the @libdir@ (expanded by the makefile) is expanded to the final install path, not the temporary installation directory used by make check. To fix, tweak things so that CREATE EXTENSION can be used instead, which solves the problem because the library path is expanded by the backend, which is aware of the true libdir. http://git.postgresql.org/pg/commitdiff/da34731bd3c805ab77d6e5914d02b1f011e9479d
  • Fix typos http://git.postgresql.org/pg/commitdiff/6597ec9be6a9ed50390f73235d6654ec32a0b944
  • doc: Fix markup. In the spirit of d34b48a021b181e30c53280d336820740f67570b Per buildfarm member guaibasaurus, via Stefan Kaltenbrunner. http://git.postgresql.org/pg/commitdiff/182362b669f484c1f92e058c710b025d25ddc5dd
  • Keep track of transaction commit timestamps. Transactions can now set their commit timestamp directly as they commit, or an external transaction commit timestamp can be fed from an outside system using the new function TransactionTreeSetCommitTsData(). This data is crash-safe, and truncated at Xid freeze point, same as pg_clog. This module is disabled by default because it causes a performance hit, but can be enabled in postgresql.conf requiring only a server restart. A new test in src/test/modules is included. Catalog version bumped due to the new subdirectory within PGDATA and a couple of new SQL functions. Authors: Álvaro Herrera and Petr Jelínek Reviewed to varying degrees by Michael Paquier, Andres Freund, Robert Haas, Amit Kapila, Fujii Masao, Jaime Casanova, Simon Riggs, Steven Singer, Peter Eisentraut http://git.postgresql.org/pg/commitdiff/73c986adde5d73a5e2555da9b5c8facedb146dcd
  • Move security_label test. Rather than have the core security_label regression test depend on the dummy_seclabel module, have that part of the test be executed by dummy_seclabel itself directly. This simplifies the testing rig a bit; in particular it should silence the problems from the MSVC buildfarm phylum, which haven't yet gotten taught how to install src/test/modules. http://git.postgresql.org/pg/commitdiff/df761e3cf79db09d602610ee61e51cb378288382

Andres Freund a poussé :

  • Don't skip SQL backends in logical decoding for visibility computation. The logical decoding patchset introduced PROC_IN_LOGICAL_DECODING flag PGXACT flag, that allows such backends to be skipped when computing the xmin horizon/snapshots. That's fine and sensible for walsenders streaming out logical changes, but not at all fine for SQL backends doing logical decoding. If the latter set that flag any change they have performed outside of logical decoding will not be regarded as visible - which e.g. can lead to that change being vacuumed away. Note that not setting the flag for SQL backends isn't particularly bothersome - the SQL backend doesn't do streaming, so it only runs for a limited amount of time. Per buildfarm member 'tick' and Alvaro. Backpatch to 9.4, where logical decoding was introduced. http://git.postgresql.org/pg/commitdiff/0fd38e13706e6bd4b2a4ce4e0570d0aa4ad65345
  • Document that pg_stat_*_tables.n_tup_upd includes n_tup_hot_upd. Author: Peter Geoghegan http://git.postgresql.org/pg/commitdiff/5ede3a31171d83b94a773d997fe05b563f8ebb3d

Peter Eisentraut a poussé :

Robert Haas a poussé :

Simon Riggs a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Jim Nasby sent in a patch to log cleanup lock acquisition failures in vacuum.
  • Tom Lane and Emre Hasegeli traded patches to add selectivity estimation for inet operators.
  • Dilip Kumar sent in another revision of a patch to allow vacuumdb to work in parallel.
  • Jeff Janes sent in a patch to attempt to fix startup/recovery problems with unlogged tables by adding an unconditional and unchecked unlink before the copy.
  • Jeff Davis sent in another revision of a patch to do better memory accounting.
  • SAWADA Masahiko and Michael Paquier traded patches to implement REINDEX SCHEMA.
  • Alexander Shulgin sent in another revision of a patch to turn recovery.conf into GUCs.
  • Craig Ringer and David Rowley traded patches to use the faster, higher precision timer API on Windows when available.
  • Teodor Sigaev sent in another revision of a patch to implement a compression method for SP-GiST.
  • Kaigai Kouhei sent in another revision of a patch to implement some interfaces in the custom scan API.
  • Peter Geoghegan sent in patches to: 1. add new abbreviation infrastructure to sortsupport, and add a single client of this new infrastructure, the text sortsupport routine. 2. Estimate total number of rows to be sorted, and 3. Alter comments to reflect current naming.
  • Matt Newell sent in two revisions of a patch to support pipelining in libpq.
  • Peter Geoghegan sent in another revision of a patch to create a Levenshtein distance column HINT for nonexistent columns.
  • Amit Kapila and David Rowley traded patches to do sequential scans in parallel.
  • Peter Geoghegan sent in a patch to clarify the documentation of pg_stat_all_tables to mention that pg_stat_*_tables.n_tup_upd includes HOT updates.
  • Heikki Linnakangas sent in another revision of a patch to implement an SSL regression test suite.
  • Michael Paquier sent in a patch to add missing descriptions for two parameters in XLOG_PARAMETER_CHANGE.
  • Petr (PJMODOS) Jelinek sent in a patch to rename action_at_recovery_target to recovery_target_action everywhere.
  • Michael Paquier sent in two more revisions of a patch to implement compression of full-pages writes.
  • Stephen Frost sent in a patch to change usage of GetUserId() to has_privs_of_role().
  • Stephen Frost sent in a patch to change the default for include_realm to zero.
  • Adam Brightwell sent in two more revisions of a patch to add a role attribute bitmask catalog representation.
  • Tomas Vondra sent in another revision of a patch to implement an adaptive ndistinct estimator.
  • Tomas Vondra sent in a patch to gracefully increasing NTUP_PER_BUCKET instead of batching.

par N Bougain le lundi 8 décembre 2014 à 22h36

lundi 1 décembre 2014

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 30 novembre 2014

Appel à conférenciers pour la PGCon 2015 lancé le 1er décembre 2014 : http://www.pgcon.org/2015/papers.php

Offres d'emplois autour de PostgreSQL en novembre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA.

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

(lien vers l'article original)

Correctifs appliqués

Heikki Linnakangas a poussé :

  • Distinguish XLOG_FPI records generated for hint-bit updates. Add a new XLOG_FPI_FOR_HINT record type, and use that for full-page images generated for hint bit updates, when checksums are enabled. The new record type is replayed exactly the same as XLOG_FPI, but allows them to be tallied separately e.g. in pg_xlogdump. http://git.postgresql.org/pg/commitdiff/0bd624d63b056205fda17a2d694d91db16468e3f
  • Add a few paragraphs to B-tree README explaining L&Y algorithm. This gives an overview of what Lehman & Yao's paper is all about, so that you can understand the rest of the README without having to read the paper. Per discussion with Peter Geoghegan and others. http://git.postgresql.org/pg/commitdiff/49b86fb1c97878ea2e3a8118df072c95f60077ac
  • Make Port->ssl_in_use available, even when built with !USE_SSL. Code that check the flag no longer need #ifdef's, which is more convenient. In particular, makes it easier to write extensions that depend on it. In the passing, modify sslinfo's ssl_is_used function to check ssl_in_use instead of the OpenSSL specific 'ssl' pointer. It doesn't make any difference currently, as sslinfo is only compiled when built with OpenSSL, but seems cleaner anyway. http://git.postgresql.org/pg/commitdiff/e453cc2741416dc784842b2bba68749556cf0f6f
  • Check return value of strdup() in libpq connection option parsing. An out-of-memory in most of these would lead to strange behavior, like connecting to a different database than intended, but some would lead to an outright segfault. Alex Shulgin and me. Backpatch to all supported versions. http://git.postgresql.org/pg/commitdiff/30bf4689a96cd283af33edcdd6b7210df3f20cd8
  • Allow "dbname" from connection string to be overridden in PQconnectDBParams If the "dbname" attribute in PQconnectDBParams contained a connection string or URI (and expand_dbname = TRUE), the database name from the connection string could not be overridden by a subsequent "dbname" keyword in the array. That was not intentional; all other options can be overridden. Furthermore, any subsequent "dbname" caused the connection string from the first dbname value to be processed again, overriding any values for the same options that were given between the connection string and the second dbname option. In the passing, clarify in the docs that only the first dbname option in the array is parsed as a connection string. Alex Shulgin. Backpatch to all supported versions. http://git.postgresql.org/pg/commitdiff/add1b052e2b2d5206474d58db25e48820242d15b
  • Allow using connection URI in primary_conninfo. The old method of appending options to the connection string didn't work if the primary_conninfo was a postgres:// style URI, instead of a traditional connection string. Use PQconnectdbParams instead. Alex Shulgin http://git.postgresql.org/pg/commitdiff/b3fc6727ce54a16ae9227bcccfebfa028ac5b16f
  • Remove dead function prototype. It was added in commit efc16ea5, but never defined. http://git.postgresql.org/pg/commitdiff/1812ee5767a25a36e7002be8a3a032357d3fe4e2
  • Fix assertion failure at end of PITR. InitXLogInsert() cannot be called in a critical section, because it allocates memory. But CreateCheckPoint() did that, when called for the end-of-recovery checkpoint by the startup process. In the passing, fix the scratch space allocation in InitXLogInsert to go to the right memory context. Also update the comment at InitXLOGAccess, which hasn't been totally accurate since hot standby was introduced (in a hot standby backend, InitXLOGAccess isn't called at backend startup). Reported by Michael Paquier http://git.postgresql.org/pg/commitdiff/afeacd274830d1e3d15ffc4e4d4b08a887f76abb

Tom Lane a poussé :

  • Allow simplification of EXISTS() subqueries containing LIMIT. The locution "EXISTS(SELECT ... LIMIT 1)" seems to be rather common among people who don't realize that the database already performs optimizations equivalent to putting LIMIT 1 in the sub-select. Unfortunately, this was actually making things worse, because it prevented us from optimizing such EXISTS clauses into semi or anti joins. Teach simplify_EXISTS_query() to suppress constant-positive LIMIT clauses. That fixes the semi/anti-join case, and may help marginally even for cases that have to be left as sub-SELECTs. Marti Raudsepp, reviewed by David Rowley http://git.postgresql.org/pg/commitdiff/b62f94c60386796fd88256c5b7b1e8301c345166
  • Support arrays as input to array_agg() and ARRAY(SELECT ...). These cases formerly failed with errors about "could not find array type for data type". Now they yield arrays of the same element type and one higher dimension. The implementation involves creating functions with API similar to the existing accumArrayResult() family. I (tgl) also extended the base family by adding an initArrayResult() function, which allows callers to avoid special-casing the zero-inputs case if they just want an empty array as result. (Not all do, so the previous calling convention remains valid.) This allowed simplifying some existing code in xml.c and plperl.c. Ali Akbar, reviewed by Pavel Stehule, significantly modified by me http://git.postgresql.org/pg/commitdiff/bac27394a1c69c20ec904729c593e59485c75c69
  • De-reserve most statement-introducing keywords in plpgsql. Add a bit of context sensitivity to plpgsql_yylex() so that it can recognize when the word it is looking at is the first word of a new statement, and if so whether it is the target of an assignment statement. When we are at start of statement and it's not an assignment, we can prefer recognizing unreserved keywords over recognizing variable names, thereby allowing most statements' initial keywords to be demoted from reserved to unreserved status. This is rather useful already (there are 15 such words that get demoted here), and what's more to the point is that future patches proposing to add new plpgsql statements can avoid objections about having to add new reserved words. The keywords BEGIN, DECLARE, FOR, FOREACH, LOOP, WHILE need to remain reserved because they can be preceded by block labels, and the logic added here doesn't understand about block labels. In principle we could probably fix that, but it would take more than one token of lookback and the benefit doesn't seem worth extra complexity. Also note I didn't de-reserve EXECUTE, because it is used in more places than just statement start. It's possible it could be de-reserved with more work, but that would be an independent fix. In passing, also de-reserve COLLATE and DEFAULT, which shouldn't have been reserved in the first place since they only need to be recognized within DECLARE sections. http://git.postgresql.org/pg/commitdiff/bb1b8f694ad2efc35ebae2acfa2c18a2197b82a1
  • Fix uninitialized-variable warning. In passing, add an Assert defending the presumption that bytes_left is positive to start with. (I'm not exactly convinced that using an unsigned type was such a bright thing here, but let's at least do this much.) http://git.postgresql.org/pg/commitdiff/d934a052348c1fa017ff677d08e74c643e3f416c
  • Free libxml2/libxslt resources in a safer order. Mark Simonetti reported that libxslt sometimes crashes for him, and that swapping xslt_process's object-freeing calls around to do them in reverse order of creation seemed to fix it. I've not reproduced the crash, but valgrind clearly shows a reference to already-freed memory, which is consistent with the idea that shutdown of the xsltTransformContext is trying to reference the already-freed stylesheet or input document. With this patch, valgrind is no longer unhappy. I have an inquiry in to see if this is a libxslt bug or if we're just abusing the library; but even if it's a library bug, we'd want to adjust our code so it doesn't fail with unpatched libraries. Back-patch to all supported branches, because we've been doing this in the wrong(?) order for a long time. http://git.postgresql.org/pg/commitdiff/c168ba311228c523923f7ef7699cba96deedcc6d
  • Improve typcache: cache negative lookup results, add invalidation logic. Previously, if the typcache had for example tried and failed to find a hash opclass for a given data type, it would nonetheless repeat the unsuccessful catalog lookup each time it was asked again. This can lead to a significant amount of useless bufmgr traffic, as in a recent report from Scott Marlowe. Like the catalog caches, typcache should be able to cache negative results. This patch arranges that by making use of separate flag bits to remember whether a particular item has been looked up, rather than treating a zero OID as an indicator that no lookup has been done. Also, install a credible invalidation mechanism, namely watching for inval events in pg_opclass. The sole advantage of the lack of negative caching was that the code would cope if operators or opclasses got added for a type mid-session; to preserve that behavior we have to be able to invalidate stale lookup results. Updates in pg_opclass should be pretty rare in production systems, so it seems sufficient to just invalidate all the dependent data whenever one happens. Adding proper invalidation also means that this code will now react sanely if an opclass is dropped mid-session. Arguably, that's a back-patchable bug fix, but in view of the lack of complaints from the field I'll refrain from back-patching. (Probably, in most cases where an opclass is dropped, the data type itself is dropped soon after, so that this misfeasance has no bad consequences.) http://git.postgresql.org/pg/commitdiff/e384ed6cdec691e0f7c9a077d0fb2a357763c335
  • Improve performance of OverrideSearchPathMatchesCurrent(). This function was initially coded on the assumption that it would not be performance-critical, but that turns out to be wrong in workloads that are heavily dependent on the speed of plpgsql functions. Speed it up by hard-coding the comparison rules, thereby avoiding palloc/pfree traffic from creating and immediately freeing an OverrideSearchPath object. Per report from Scott Marlowe. http://git.postgresql.org/pg/commitdiff/96d66bcfc60d9bcb7db767f23d33abf4d8bc7021
  • Add bms_next_member(), and use it where appropriate. This patch adds a way of iterating through the members of a bitmapset nondestructively, unlike the old way with bms_first_member(). While bms_next_member() is very slightly slower than bms_first_member() (at least for typical-size bitmapsets), eliminating the need to palloc and pfree a temporary copy of the target bitmapset is a significant win. So this method should be preferred in all cases where a temporary copy would be necessary. Tom Lane, with suggestions from Dean Rasheed and David Rowley http://git.postgresql.org/pg/commitdiff/f4e031c662a6b600b786c4849968a099c58fcce7
  • Add bms_get_singleton_member(), and use it where appropriate. This patch adds a function that replaces a bms_membership() test followed by a bms_singleton_member() call, performing both the test and the extraction of a singleton set's member in one scan of the bitmapset. The performance advantage over the old way is probably minimal in current usage, but it seems worthwhile on notational grounds anyway. David Rowley http://git.postgresql.org/pg/commitdiff/d25367ec4f869aac80e97964fa5d7143536818b1
  • Fix minor bugs in commit 30bf4689a96cd283af33edcdd6b7210df3f20cd8 et al. Coverity complained that the "else" added to fillPGconn() was unreachable, which it was. Remove the dead code. In passing, rearrange the tests so as not to bother trying to fetch values for options that can't be assigned. Pre-9.3 did not have that issue, but it did have a "return" that should be "goto oom_error" to ensure that a suitable error message gets filled in. http://git.postgresql.org/pg/commitdiff/1adbb347ec6ca059168a1866357ca7a23d117272

Stephen Frost a poussé :

  • Suppress DROP CASCADE notices in regression tests. In the regression tests, when doing cascaded drops, we need to suppress the notices from DROP CASCADE or there can be transient regression failures as the order of drops can depend on the physical row order in pg_depend. Report and fix suggestion from Tom. http://git.postgresql.org/pg/commitdiff/81d815dc3ed74a7d20f7b4c7d87c42816c7ee7af
  • Add int64 -> int8 mapping to genbki. Per discussion with Tom and Andrew, 64bit integers are no longer a problem for the catalogs, so go ahead and add the mapping from the C int64 type to the int8 SQL identification to allow using them. Patch by Adam Brightwell http://git.postgresql.org/pg/commitdiff/25976710dfd8611d3fc79c0c1e20179ff7a940ec
  • Rename pg_rowsecurity -> pg_policy and other fixes As pointed out by Robert, we should really have named pg_rowsecurity pg_policy, as the objects stored in that catalog are policies. This patch fixes that and updates the column names to start with 'pol' to match the new catalog name. The security consideration for COPY with row level security, also pointed out by Robert, has also been addressed by remembering and re-checking the OID of the relation initially referenced during COPY processing, to make sure it hasn't changed under us by the time we finish planning out the query which has been built. Robert and Alvaro also commented on missing OCLASS and OBJECT entries for POLICY (formerly ROWSECURITY or POLICY, depending) in various places. This patch fixes that too, which also happens to add the ability to COMMENT on policies. In passing, attempt to improve the consistency of messages, comments, and documentation as well. This removes various incarnations of 'row-security', 'row-level security', 'Row-security', etc, in favor of 'policy', 'row level security' or 'row_security' as appropriate. Happy Thanksgiving! http://git.postgresql.org/pg/commitdiff/143b39c1855f8a22f474f20354ee5ee5d2f4d266

Simon Riggs a poussé :

Robert Haas a poussé :

Peter Eisentraut a poussé :

Fujii Masao a poussé :

Álvaro Herrera a poussé :

Noah Misch a poussé :

Correctifs rejetés (à ce jour)

  • No one was disappointed this week

Correctifs en attente

  • Dilip Kumar sent in another revision of a patch to allow doing vacuumdb in parallel.
  • Amit Langote sent in a patch to fix a typo in a comment in src/bin/pg_dump.c.
  • Álvaro Herrera sent in another revision of a patch to track commit timestamps.
  • Michael Paquier and Rahila Syed traded patches to improve compression of full-page writes.
  • Adam Brightwell sent in two more revisions of a patch to implement role attribute bitmask catalog representation.
  • Ali Akbar sent in another revision of a patch to fix xpath() to return namespace definitions.
  • Marius Timmer sent in a patch to add an option to EXPLAIN to include sort order information in VERBOSE mode.
  • Michael Paquier sent in two revisions of a patch to allow compiling C++ extensions on MSVC using scripts in src/tools.
  • Ants Aasma sent in a patch to use RTLD_DEEPBIND to handle symbol conflicts in loaded libraries.
  • Pavel Stehule sent in another revision of a patch to add an ASSERT statement to PL/pgsql.
  • Peter Geoghegan sent in another revision of a patch to implement INSERT ... ON CONFLICT {IGNORE | UPDATE}.
  • Alexander Shulgin sent in another revision of a patch to add an 'ssl_protocols' configuration option.
  • Álvaro Herrera sent in a flock of patches to move test modules from contrib to src/test/modules.
  • Mats Erik Andersson sent in a patch to improve error recovery in initdb when the password file is empty.
  • Mats Erik Andersson sent in a patch to localize the text response of pg_isready.
  • Ian Lawrence Barwick sent in a patch to implement DDL deparsing.
  • Michael Paquier sent in another revision of a patch to implement fillfactor for GIN indexes.
  • Alexander Shulgin sent in a patch to allow TOAST tables to be moved to a different tablespace.
  • Peter Eisentraut sent in a patch to allow using Core Foundation locale functions on OSX.
  • Tom Lane sent in a patch to fix some misleading error messages in json_to_record.
  • Michael Paquier sent in a patch to add a new error code to track unsupported contexts.
  • Tomas Vondra sent in another revision of a patch to lower array_agg's memory requirements.
  • Stephen Frost sent in another revision of a patch to clean up GetUserId() for pg_stat and pg_signal.
  • Noah Misch sent in another revision of a patch to help secure "make check".
  • David Rowley sent in a patch to allow removing inner JOINs in some cases.
  • Andrew Dunstan sent in a patch to fix the check for whether something is considered a number in JSON.
  • Ian Lawrence Barwick sent in a patch to fix a typo/spacing problem in doc/src/sgml/wal.sgml.

par N Bougain le lundi 1 décembre 2014 à 23h03

jeudi 2 octobre 2014

Dimitri Fontaine

Meetup PostgreSQL à Paris

Mercredi 8 octobre se tiendra le prochain Meetup PostgreSQL à Paris dans les locaux de Mozilla Europe, dont la capacité est de 90 personnes ! Venez nombreux !

Le programme de cette édition est un peu particulier puisque nous avons reçu quatre propositions enthousiastes de présentations. Chacun aura donc 15 à 20 minutes (questions incluses) pour présenter son sujet. Voici le programme :

  • Aggrégation temporelle sous contrainte d'iops, par Jean-Gérard Pailloncy
  • Le partitionnement de tables démystifié, par Flavio Henrique Araque Gurgel
  • PoWA, par Thomas Reiss et Julien Rouhaud
  • PostgreSQL pour les développeurs, par Dimitri Fontaine

Nous aurons aussi le plaisir de partager un buffet sponsorisé cette fois par Novapost, un grand merci à eux.

À mercredi !

par dim@tapoueh.org (Dimitri Fontaine) le jeudi 2 octobre 2014 à 09h49

lundi 22 septembre 2014

Nicolas Thauvin

Confiner PostgreSQL avec SELinux

J'avais déjà expérimenté un peu avec SELinux il y a deux ans sans aller trop loin, parce que j'entendais souvent la phrase "Si on veut de la sécurité, il faut SELinux" et surtout à cause de l'arrivée de l'extension sepgsql dans les modules contrib de PostgreSQL. Ça avait donné une conf pour le Fosdem où finalement, j'ai plus parlé des privilèges classiques que de SELinux.

Suite à une demande au ${BOULOT}, je me suis replongé dedans, et les choses ont peu évolué. Dans la majorité des recherches que j'ai pu faire, SELinux reste tout de même un truc qui se met en travers la route, c'est-à-dire, qu'il y a toujours plus de résultats sur comment le désactiver plutôt que configurer les choses correctement. Ensuite, certains proposent de faire aveuglement confiance à setroubleshoot et audit2allow pour faire taire la bête. Enfin, j'ai dû trouver une page ou deux après en deux semaines à temps plein sur le sujet qui parlent de comment confiner un utilisateur dans le but d'implémenter ce que promet SELinux et que toutes les entreprises demandent : le RBAC, Role Based Access Control, ou comment donner le moins de droits possibles aux sous-traitants qui hébergent les serveurs.

Cette fois ci, je suis allé plus loin. Déjà, je suis parti sur une installation de CentOS 6, la famille de distribution RHEL/CentOS/Fedora semble être la plus en avance par rapport à SELinux : à peu près tous les services/daemons fournis dans l'install sont confinés par SELinux. Les utilisateurs ne le sont pas au login, il n'y a donc aucune configuration de RBAC. On verra peut-être ça plus tard, j'ai pas encore obtenu de résultat satisfaisant sur ce sujet, et il vaut mieux expliquer comment confiner correctement PostgreSQL avant de passer aux roles. Tout simplement, parce qu'il est très simple de casser ce confinement avec la configuration par défaut de RHEL, grâce à pg_ctl. Enfin, l'installation des paquets du PGDG n'est pas confinée par défaut, il manque les file contexts adaptés aux chemins particuliers de ces paquets, prévu pour faire cohabiter plusieurs versions majeures ; ce qui ne correspond pas ce qu'à prévu Red Hat pour PostgreSQL.

Après cette longue introduction, mais avant de commencer, il faut savoir administrer un minumum SELinux : si vous ne savez pas qu'il existe des options -Z, ce que sont les contexts, les types et les domaines, mieux vaut d'abord se documenter, par exemple chez Red Hat.

Voici donc les file contexts à ajouter dans un fichier module.te pour confiner l'installation PostgreSQL du PGDG :

/etc/rc\.d/init\.d/(se)?postgresql(-.*)?    --  gen_context(system_u:object_r:postgresql_initrc_exec_t,s0)
/usr/pgsql-[0-9]+\.[0-9]+/bin/initdb        --  gen_context(system_u:object_r:postgresql_exec_t,s0)
/usr/pgsql-[0-9]+\.[0-9]+/bin/pg_ctl        --  gen_context(system_u:object_r:postgresql_initrc_exec_t,s0)
/usr/pgsql-[0-9]+\.[0-9]+/bin/postgres      --  gen_context(system_u:object_r:postgresql_exec_t,s0)
/usr/pgsql-[0-9]+.[0-9]+/share/locale(/.*)?     gen_context(system_u:object_r:locale_t,s0)
/usr/pgsql-[0-9]+.[0-9]+/share/man(/.*)?        gen_context(system_u:object_r:man_t,s0)

Tout d'abord, pour l'init script et pg_ctl, on utilise le type postgresql_initrc_exec_t, c'est ce qui permet de lancer PostgreSQL dans le domaine confiné postgresql_t au boot, via l'init script, et manuellement. La méthode la plus propre est de toujours utiliser l'init script, idéalement par l'intermédiaire de run_init pour démarrer, arrêter ou redémarrer le postmaster. On évite alors de laisser trainer des choses dans /var/{run,lock}.

Les programmes postgres et initdb doivent avoir le type postgresql_exec_t car ils exécutent le serveur PostgreSQL ; cela doit se faire dans le domaine postgresql_t.

Enfin, on a placé les labels corrects sur les fichiers de traduction et les pages de man, pour faire plus propre. Ce code source de module SELinux alors doit être compilé et chargé.

Cette configuration est reprise dans le module SELinux disponible sur github. On peut aussi l'ajouter manuellement avec semanage :

semanage fcontext -a -t postgresql_initrc_exec_t '/etc/rc\.d/init\.d/(se)?postgresql(-.*)?'
semanage fcontext -a -t postgresql_exec_t '/usr/pgsql-[0-9]+\.[0-9]+/bin/initdb'
semanage fcontext -a -t postgresql_initrc_exec_t '/usr/pgsql-[0-9]+\.[0-9]+/bin/pg_ctl'
semanage fcontext -a -t postgresql_exec_t '/usr/pgsql-[0-9]+\.[0-9]+/bin/postgres'
semanage fcontext -a -t locale_t '/usr/pgsql-[0-9]+.[0-9]+/share/locale(/.*)?'
semanage fcontext -a -t man_t '/usr/pgsql-[0-9]+.[0-9]+/share/man(/.*)?'

Il existe un certain nombre de booleans pour la configuration des droits SELinux de PostgreSQL, le plus important concerne rsync, souvent nécessaire pour faire des base backups. Il s'agit de postgresql_can_rsync, pour l'activer :

semanage boolean -m --on postgresql_can_rsync

Si on lance l'instance sur un port TCP différent de 5432, il faut l'autoriser dans la configuration locale de SELinux :

semanage port -a -t postgresql_port_t -p tcp <port>

Enfin, il ne faut pas oublier d'appliquer les contexts aux fichiers soit avec restorecon, un relabel complet au reboot ou chcon.

lundi 22 septembre 2014 à 14h23

Rodolphe Quiédeville

Django Paginator oui mais

Après la lecture de l'excellent article de Markus Winnand no-offset j'ai réalisé un test pour mesurer la différence de performance de l'utilisation d'OFFSET avec Django.

La suite à lire sur le blog Novapost's paradize.

par Rodolphe Quiédeville le lundi 22 septembre 2014 à 08h13

samedi 12 avril 2014

Guillaume Lelarge

Nouvelle série d'articles pour GLMF

Après l'article sur les nouveautés de la version 9.3, j'ai entamé une série d'articles sur le fonctionnement du planificateur (aussi appelé optimiseur).

Le premier est sorti ce mois-ci, dans le numéro 170 de GNU/Linux Magazine France. Il traite des différents types de parcours que l'optimiseur peut planifier.

J'ai loupé le coche pour le prochain numéro, donc celui sur les jointures devrait sortir dans le numéro 172 si tout va bien. En fait, je viens tout juste de finir son écriture.

Il y aura certainement deux autres articles, un sur les autres types de nœuds et un sur les outils pour la compréhension des plans d'exécution mais ils restent à écrire.

En attendant, je suis preneur de toute remarque/critique sur mes articles :)

par Guillaume Lelarge le samedi 12 avril 2014 à 16h11

mardi 11 mars 2014

Rodolphe Quiédeville

Debian, PG9.3, Osmosis et Nominatim

Même en se basant sur des références en terme de stabilité il peut arriver que certains combos soient fatals à votre production. C'est ce qui m'est arrivé récemment pour un serveur Nominatim installé pourtant sur une Debian Wheezy, osmosis est utilisé pour la mise à jour continue de la base Nominatim et m'a fait des misères que je m'en vais vous conter.

En parallèle de la version de PostgreSQL 9.1 standard Debian j'ai installé sur la machine une 9.3 en provenance du dépôt Debian de la communauté PG (le support dfe JSON dans PG 9.3 c'est awesome), jusque là tout va bien. Seulement à l'upgrade suivant le paquet libpostgis-java est passé en version 2.1.1-5.pgdg70+1 (celle-ci étant disponible sur le dépôt PG) ; malheureusement cette version est incompatible avec osmosis packagé chez Debian qui nécessite la version 1.5.3 de cette librairie, et là c'est le drâme !

Donc si au lancement d'osmosis vous rencontrez l'erreur :

java.io.FileNotFoundException: /usr/share/java/postgis.jar

Alors que le sus-dit fichier est bien présent, et que vous commencez à dire du mal des backtraces java, il existe un solution simple et efficace, downgrader la version de libpostgis-java en quelques commandes :

dpkg --remove osmosis
apt-get install libpostgis-java=1.5.3-2 osmosis

Sans oublier un pinning pour éviter la future mise à jour du paquet.

par Rodolphe Quiédeville le mardi 11 mars 2014 à 08h07

dimanche 2 mars 2014

Guillaume Lelarge

Deux news... (GLMF + traduction)

Ça fait un bon moment que je n'ai pas publié un billet sur ce blog... ça fait peur :)

Bref, deux nouvelles intéressantes.

J'ai mis à jour les manuels français suite aux dernières versions mineures. Ça n'a pas été spécialement long de le faire. Par contre, il n'a pas été simple de trouver le temps pour le faire. Mais bon, c'est fait, les manuels sont à jour.

Il y avait aussi longtemps que je n'avais pas écrit un article pour le GNU/Linux Magazine France. J'ai enfin repris, avec un article sur les nouveautés de la version 9.3. Il est paru sur le GLMF 169. Je pense qu'il y aura d'autres articles, suivant le temps à ma disposition et la motivation que j'ai. Pour l'instant, j'essaie d'écrire sur le planificateur de requêtes, sujet que j'étudie depuis plus d'un an maintenant. Cela étant dit, si vous avez des idées de sujets, je suis preneur :)

par Guillaume Lelarge le dimanche 2 mars 2014 à 21h54