PostgreSQL La base de donnees la plus sophistiquee au monde.

La planete francophone de PostgreSQL

mardi 14 février 2017

Loxodata

PostgreSQL pour les DBAs Oracle

En tant que DBA Oracle, il n’est pas très compliqué de monter en compétences sur PostgreSQL. Les deux moteurs sont assez proches l’un de l’autre par leur fonctionnement.
Cependant, le vocabulaire n’est pas toujours le même.

Cet article fait le point sur les différences entre Oracle et PostgreSQL.

Fonctionnalités

Sauvegarde/Restauration (RMAN)

Cette fonctionnalité est incluse dans la licence Oracle, mais si on souhaite stocker le catalogue RMAN dans une base, il faudra une licence pour cette base.
PostgreSQL propose un utilitaire permettant de gérer les sauvegardes : pg_basebackup. Il existe plusieurs contributions sur le sujet. Vous pouvez notamment regarder pgBarman (pour les nostalgiques d’RMAN) ou pgBackRest.

Dump logique des données (exp/expdp/imp/impdp)

Ces outils sont inclus dans les licences Oracle.
PostgreSQL met à disposition deux outils natifs pg_dump et pg_restore. Il est à noter que vous ne risquez pas de problème de cohérence de données lors du dump car PostgreSQL n’exporte que des données cohérentes entre elles. (Que celui qui n’a jamais rencontré de “ORA-01555 Snapshot too old” me jette la première pierre.)

Gestion du stockage (ASM)

ASM est inclus dans les licences Oracle, cependant, si on souhaite certaines fonctionnalités (snapshot, sécurité, réplication, audit), il faudra payer des fonctionnalités supplémentaires.
PostgreSQL ne gère pas le stockage. Il laisse l’OS faire son travail, car c’est le sien.

Réplication (DataGuard)

DataGuard nécessite une licence Enterprise par nœud.
PostgreSQL permet une réplication physique équivalente au DataGuard, soit par log-shipping (Warm standby), soit par streaming (équivalent d’Oracle Streams -> Hot Standby). La standby peut être ouverte en lecture seule ou pas.
La réplication logique existe aussi sous PostgreSQL. Il s’agit de la contribution pgLogical. Elle sera intégrée définitivement à PostgreSQL dans la version 10.

Haute disponibilité (Dataguard avec Observer)

Active DataGuard nécessite une option payante en plus d’une licence Enterprise par nœud.
PostgreSQL ne propose pas en natif de système comme l’« observer », mais ce type de solution est implémentée côté OS (voir Heartbeat, Pacemaker ou PowerHA). Ces systèmes sont très fiables et performants.

Réplication multi-maître

La réplication multi-maître Oracle nécessite une Licence Directory Server de niveau enterprise en plus des licences Oracle pour chaque base.
PostgreSQL n’inclue pas en natif de réplication multi-maîtres, mais la contribution BDR le permet.

Réplication logique multi-technologies (GoldenGate)

GoldenGate est un produit à part chez Oracle, il faudra donc acquérir la licence Golden Gate en plus des licences pour les bases.
Il est posssible de faire de la réplication logique entre différentes technologies comme avec pg_chameleon qui permet par exemple une réplication logique entre mySQL et PostgreSQL.
BDR propose une réplication logique multi-maître.

Sharding (Oracle 12c)

Le sharding sous Oracle nécessite des licences Enterprise.
Il est possible de faire du sharding avec PostgreSQL (Postgres XL ou projet Atomic). L’extension PostgreSQL Citus-data le permet également.

Plusieurs instances pour servir la même base de données (RAC)

Pour mettre en place un RAC, il faut compter une licence Enterprise par noœud ainsi que l’option payante RAC.
Il n’existe pas à proprement parler d’équivalent à RAC sous PostgreSQL. Cependant, il est possible de faire du load balancing avec la contribution pgPool. De plus, les derniers drivers JDBC intègrent le load balancing.

Bases géographiques (Oracle spatial)

Oracle Spatial est une option payante de la version Enterprise.
PostgreSQL dispose d’une extension, PostGIS, qui gère très bien les données géographiques. PostGIS est la référence en matière de cartouche SIG.

Fonctionnement interne du moteur

Instances et bases de données

Sous Oracle, en standard, une instance ne peut servir qu’une seule base de données. Il est possible d’aller au-delà de cette limitation en prenant l’option payante Multitenant de la version Enterprise.
Sous PostgreSQL, une instance sert plusieurs bases de données. La norme SQL définit cette notion sous le nom groupe de cataloge (catalog cluster). Sous PostgreSQL, on utilise le terme de “groupe de bases de données” (database cluster). Bien sûr, ce terme n’a aucun rapport avec le RAC d’Oracle.

Processus d’écoute (Listener)

PostgreSQL ne travaille pas avec un process séparé pour l’écoute des connexions distantes. Le process “maître” postgres assure ce travail.

Tablespace

En Oracle, un tablespace est une sorte de partition logique qui comporte un ou des fichier(s) de données. Sous PostgreSQL, un tablespace est un répertoire sur le système de fichier. Il n’y a pas de taille limite pour un tablespace. (En dehors de celles imposées par le système d’exploitation, évidemment.)

Il est possible de partitionner une table et de répartir ses partitions sur différents tablespaces.
Il est possible, comme avec Oracle, de définir un tablespace différent pour un index.

Tablespace Undo

Le tablespace undo est intégré dans le fichier correspondant à chaque table. Cet espace est bien sûr réutilisé comme le tablespace undo. Cependant, pour récupérer cet espace, un processus de maintenance, le vacuum, est lancé régulièrement.

Tablespace temporaire

Sous Oracle, certaines opérations sont effectuées dans le tablespace temporaire. PostgreSQL utilise un répertoire nommé pgsql_tmp au sein du tablespace par défaut de la base de données. Il est possible de définir plusieurs tablespaces temporaires.

Fichiers de données et tables TOAST

Les données d’une table sont stockées dans un fichier de données d’1Go dans son tablespace. Si la taille de la table devait dépasser le giga octet, plusieurs fichiers seront créés.
Il n’est pas possible de mettre les données d’une colonne dans un tablespace spécifique (comme on le fait souvent avec des données LOB sous Oracle). Cependant, losrqu’une ligne de données dépasse les 2 Ko (valeur par défaut modifiable), PostgreSQL appelle automatiquement un système nommé TOAST (The Oversized-Attribute Storage Technique) pour stocker ces données dans un fichier séparé.

Schémas, Rôles et Users

Sous Oracle, un user est lié à un schéma qui correspond à son espace personnel d’objets. Un schéma n’est pas un objet en lui-même chez Oracle, on ne peut donc pas donner des droits sur un schéma. Un rôle permet de créer des droits génériques à appliquer à un user.
Sous PostgreSQL, on peut utiliser le DDL CREATE USER, mais il ne va faire qu’appeler le DDL CREATE ROLE en ajoutant les droits de connexion.
Un schéma est totalement indépendant d’un user. C’est une sorte de conteneur d’objets. On peut donc donner des droits sur un schéma et par héritage sur tous les objets de ce schéma. Par contre, le user peut définir un chemin par défaut (qui peut contenir plusieurs schémas) pour rechercher ses objets.

Table dual

Il n’existe pas de table dual, conformément à la norme SQL qui autorise les requêtes SELECT sans précision de la clause FROM.
Exemple :

SELECT fonction();
SELECT 2+2;

Petite correspondance rapide des fichiers Oracle

.tg {border-collapse:collapse;border-spacing:0;} .tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;} .tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;} .tg .tg-yw4l{vertical-align:top}

Oracle
PostgreSQL
Commentaire
Parameter file pg_hba.conf
pg_ident.conf
postgresql.conf
Les fichiers pg_hba.conf et pg_ident.conf permettent de gérer le mode d’identification et l’authentification.
Password file
postgres DB
Control files
postgres DB
Online redo
pg_xlog
(ou WAL)
A partir de la version 10, pour éviter les suppressions intempestives, le répertoire pg_xlog sera renommé en pg_wal. Wal signifie Write Ahead Log.
Archived logs
pg_xlog
(ou WAL)
Il n’y a pas de nom spécifique pour les WAL archivés
Data files
$PGDATA
Alert log
Server log
Il est possible de configurer l’instance pour voir les logs de connexion (qu’on verrait chez Oracle dans le fichier listener.log).

par contact@loxodata.com (Loxodata) le mardi 14 février 2017 à 14h53

jeudi 9 février 2017

Loxodata

PostgreSQL 9.6.2 et autres correctifs

Publication de PostgreSQL 9.6.2 et autres mises à jour cumulatives

Le PostgreSQL Global Development Group a publié une mise à jour de toutes les versions supportées de notre système de base de données.
Il s’agit des versions mineures 9.6.2, 9.5.6, 9.4.11, 9.3.16 et 9.2.20.
Cette publication corrige des problèmes potentiels de corruptions de données lors de la construction d’index et dans certaines situations d’applications des journaux de transactions (write-ahead-log).
Ces situations sont détaillées ci-dessous.
Elle corrige aussi plus de 75 bugs reportés ces 3 derniers mois.

Comme toujours, les mises à jour doivent être appliquées le plus rapidement possible.

Corruption due à CREATE INDEX CONCURRENTLY

Il existe une situation de concurrence (« race condition ») si CREATE INDEX CONCURRENTLY est appelé sur une colonne qui n’a pas déjà été indexée.
Il se peut que des enregistrements insérées ou mis à jour par des transactions exécutées en même temps que la commande CREATE INDEX CONCURRENTLY soient indexés incorrectement.

Si un tel cas est suspecté, la solution la plus fiable consiste à reconstruire les index concernés après l’installation de la mise à jour.

Ce problème est présent dans les versions 9.2, 9.3, 9.4, 9.5, et 9.6 de PostgreSQL.

Correctifs de visibilité et stabilité de l’enregistrement des journaux de transactions

Cette publication contient plusieurs correctifs améliorant la stabilité des données visibles et les enregistrements des journaux de transactions (WAL) que nous souhaitons mettre en évidence ici.

Avant cette publication, des données pouvaient être prématurément supprimées par une opération de maintenance (« vacuum ») lorsqu’un « snapshot » particulier, utilisé pour les lectures du catalogue, est disponible.
Plus particulièrement, le « vacuum » n’a pas connaissance du plus vieux « xmin » de ce « snapshot » spécial. L’erreur apparaîtra avec un message du style :

    “cache lookup failed for relation 1255”

Cette mise à jour s’assure que les « vacuum » prennent en compte les « snapshots » de lectures du catalogue.

Il y a égalament plusieurs correctifs améliorant la stabilité de l’enregistrement des journaux de transactions, dont :

  • un correctif pour l’enregistrement dans les journaux de transactions des index BRIN où une application pouvait rendre une portion de l’index BRIN inutile et nécessiter un nouveau calcul ;
  • un correctif à propos des tables non-tracées (« unlogged ») où un enregistrement dans les journaux de transactions était créé, avec le paramètre wal_level = minimal, mais lors d’un rejeu après crash, la table semblait n’avoir pas été correctement réinitialisée ;
  • un correctif dans la validation de l’entête d’une page des journaux de transactions (« WAL »), lors de la relecture des segments, corrigeant l’erreur “out-of-sequence TLI” pouvant être reportée pendant la phase de restauration.

Ces problèmes sont présents dans la version 9.6 de PostgreSQL et peuvent être présents dans les versions 9.2, 9.3, 9.4 et 9.5.

Correctifs et améliorations

Cette mise à jour corrige un certain nombre de bugs rapportés au cours des derniers mois. Certains de ces problèmes affectent uniquement la version 9.6, mais la plupart affectent toutes les versions supportées.
Il y a plus de 75 correctifs fournis dans cette publication, dont :

  • plusieurs correctifs pour le fonctionnement en mode « hot standby » ;
  • interdication du réglage num_sync à zéro dans synchronous_standby_names ;
  • ne pas comparer le nombre de processus d’arrière-plan et la limite de connexions d’un utilisateur ;
  • correction de la vérification de la possibilité de suppression d’un objet membre d’une extension ;
  • correction du suivi des privilèges initiaux des objets membres d’une extension afin de faire fonctionner correctement ALTER EXTENSION … ADD/DROP ;
  • plusieurs correctifs sur vacuum et autovacuum ;
  • correction de CREATE OR REPLACE VIEW pour mettre à jour la requête de la vue avant d’essayer d’appliquer les nouvelles options de la vue ;
  • s’assurer qu’un ALTER TABLE préserve l’assignation du tablespace d’un index lors de la reconstruction des index ;
  • plusieurs correctifs sur le planificateur de requêtes, incluant des correctifs sur les tables étrangères et les CTEs ;
  • plusieurs correctifs autour de la recherche plein-texte pour améliorer la pertinence et les performances de la recherche ;
  • plusieurs correctifs et améliorations de performance des fonctions « tableaux » (« array functions ») ;
  • plusieurs correctifs d’interactions entre les contraintes de clés étrangères et les fonctions déclencheurs autour d’opérations spécifiques d’ALTER TABLE ;
  • suppression d’optimisations des types de données date et heure retournant des données incorrectes ;
  • correction de l’usage incorrect de la vue reloptions en tant que table régulière reloptions ;
  • correction du message incorrect « target lists can have at most N entries » lors de l’utilisation de ON CONFLICT avec de grandes tables ;
  • correction de la fausse erreur « query provides a value for a dropped column » pendant l’utilisation de INSERT ou UPDATE sur une table ayant une colonne supprimée ;
  • interdiction de l’expansion multi-colonne de foo.* dans une expression source d’un UPDATE ;
  • assurance que le typmods d’une colonne est déterminé avec précision pour les constructions VALUES multi-colonnes ;
  • plusieurs correctifs pour l’outil en ligne de commande psql ;
  • interdiction de l’exécution concurrente de plusieurs appels de pg_start_backup() et pg_stop_backup() ;
  • plusieurs correctifs pour pg_dump, pg_restore et pg_basebackup, incluant un possible échec de pg_basebackup sur un serveur standby incluant les journaux de transactions ;
  • plusieurs correctifs pour les tâches de parallélisation et les plans d’exécution des requêtes parallèles, incluant la correction d’une interruption de service si le nombre de tâches disponibles pour une requête parallèle décroît pendant une relecture ;
  • plusieurs correctifs à PL/pgSQL, PL/Python, et PL/Tcl ;
  • plusieurs correctifs des modules contrib ;
  • permettre les terminateurs de ligne de style DOS dans le fichier ~/.pgpass, même sur Unix.

Cette mise à jour contient également la version 2016j de tzdata avec les modifications de loi DST en Chypre du nord (ajout d’une nouvelle zone Asia/Famagusta), Russie (ajout d’une nouvelle zone Europe/Saratov), Tonga, et Antarctica/Casey. Corrections historiques pour l’Italie, Kazakhstan, Malte, et la Palestine. Bascule vers l’utilisation préférentielle de l’abréviation numérique pour Tonga.

Mise à jour

Toutes les mises à jours de PostgreSQL sont cumulatives.
Comme pour toutes les mises à jours, il n’est pas nécessaire d’effectuer un export/import des bases ou d’utiliser pg_upgrade pour appliquer la mise à jour courante. Il suffit d’arrêter PostgreSQL et de mettre à jour les binaires.

Si vous pensez avoir été affecté par le bug CREATE INDEX CONCURRENTLY mentionné ci-dessus, vous aurez à reconstruire l’index.
Exemple de reconstruction d’index sans perdre la possibilité de l’utiliser :

    CREATE INDEX CONCURRENTLY new_index_name ON table_name (column_name);
    DROP INDEX CONCURRENTLY old_index_name;
    ALTER INDEX new_index_name RENAME TO old_index_name;

Il est à noter que cette méthode implique la présence temporaire de deux copies de l’index. Si l’espace disque pose problème, il faut utiliser une autre méthode.

Les utilisateurs qui ont manqué une ou plusieurs mises à jour devront éventuellement effectuer des opérations post-mise à jour supplémentaires. Voir les notes de version à ce propos.

Liens

par contact@loxodata.com (Loxodata) le jeudi 9 février 2017 à 14h00

mercredi 8 février 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 5 février 2017

Le premier pgDay Israel 2017 aura lieu le 2 mars 2017. Les inscriptions sont ouvertes : http://pgday.org.il/

Postgres Vision aura lieu à Boston du 26 au 28 juin 2017. L'appel à conférenciers court jusqu'au 24 février 2017 : écrire à CFP AT PostgresVision DOT com : http://postgresvision.com/

[ndt: Meetup à Nantes le 8 mars :https://www.meetup.com/fr-FR/PostgreSQL-User-Group-Nantes/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en février

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170205221731.GA32585@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Stephen Frost pushed:

  • Handle ALTER EXTENSION ADD/DROP with pg_init_privs. In commit 6c268df, pg_init_privs was added to track the initial privileges of catalog objects and extensions. Unfortunately, that commit didn't include understanding of ALTER EXTENSION ADD/DROP, which allows the objects associated with an extension to be changed after the initial CREATE EXTENSION script has been run. The result of this meant that ACLs for objects added through ALTER EXTENSION ADD were not recorded into pg_init_privs and we would end up including those ACLs in pg_dump when we shouldn't have. This commit corrects that by making sure to have pg_init_privs updated when ALTER EXTENSION ADD/DROP is run, recording the permissions as they are at ALTER EXTENSION ADD time, and removing any if/when ALTER EXTENSION DROP is called. This issue was pointed out by Moshe Jacobson as commentary on bug #14456 (which was actually a bug about versions prior to 9.6 not handling custom ACLs on extensions correctly, an issue now addressed with pg_init_privs in 9.6). Back-patch to 9.6 where pg_init_privs was introduced. http://git.postgresql.org/pg/commitdiff/e54f75722c720b596ec5e72154cc899da199de5b
  • test_pg_dump TAP test whitespace cleanup. The formatting of the perl hashes used in the TAP tests for test_pg_dump was rather horribly inconsistent and made it more difficult than it really should have been to add new tests or adjust what tests are for what runs, etc. Reformat to clean that all up. Whitespace-only changes. http://git.postgresql.org/pg/commitdiff/fb94ca77f1400e236b00d432dccfbe4f1124971c
  • test_pg_dump: perltidy cleanup. As pointed out by Alvaro, we actually use perltidy on the perl scripts in the source tree, so go back to the results of a perltidy run for the test_pg_dump TAP script. To make it look slightly less tragic, I changed most of the independent arguments into long-form single arguments (eg: -f file.sql changed to be --file=file.sql) to avoid having them confusingly split across lines due to perltidy. Back-patch to 9.6, as the last patch was. http://git.postgresql.org/pg/commitdiff/58da8334308c26107ebb7ee06c99589e14bd882b
  • perltidy pg_dump TAP tests. The pg_dump TAP tests have gotten pretty far from what perltidy thinks they should be, so fix that, and in passing use long-form argument names with arguments passed via "=" in a similar vein to 58da833. No functional changes here, just whitespace and changing runs from "-f" to "--file=", and similar. http://git.postgresql.org/pg/commitdiff/6af8b89adba16f97bee0d3b01256861e10d0e4f1
  • pg_dump: Fix handling of ALTER DEFAULT PRIVILEGES. In commit 23f34fa, we changed how ACLs were handled to use the new pg_init_privs catalog and to dump out the ACL commands as REVOKE+GRANT combinations instead of trying to REVOKE all rights always and then GRANT back just the ones which were in place. Unfortunately, the DEFAULT PRIVILEGES system didn't quite get the correct treatment with this change and ended up (incorrectly) only including positive GRANTs instead of both the REVOKEs and GRANTs necessary to preserve the correct privileges. There are only a couple cases where such REVOKEs are possible because, generally speaking, there's few rights which exist on objects by default to be revoked. Examples of REVOKEs which weren't being correctly preserved are when privileges are REVOKE'd from the creator/owner, like so: ALTER DEFAULT PRIVILEGES FOR ROLE myrole REVOKE SELECT ON TABLES FROM myrole; or when other default privileges are being revoked, such as EXECUTE rights granted to public for functions: ALTER DEFAULT PRIVILEGES FOR ROLE myrole REVOKE EXECUTE ON FUNCTIONS FROM PUBLIC; Fix this by correctly working out what the correct REVOKE statements are (if any) and dump them out, just as we do for everything else. Noticed while developing additional regression tests for pg_dump, which will be landing shortly. Back-patch to 9.6 where the bug was introduced. http://git.postgresql.org/pg/commitdiff/e2090d9d20d8091c9478a674d9c22fc8006909ce

Heikki Linnakangas pushed:

Tom Lane pushed:

  • Update time zone data files to tzdata release 2016j. DST law changes in northern Cyprus (new zone Asia/Famagusta), Russia (new zone Europe/Saratov), Tonga, Antarctica/Casey. Historical corrections for Asia/Aqtau, Asia/Atyrau, Asia/Gaza, Asia/Hebron, Italy, Malta. Replace invented zone abbreviation "TOT" for Tonga with numeric UTC offset; but as in the past, we'll keep accepting "TOT" for input. http://git.postgresql.org/pg/commitdiff/308d8682740391cf6e94fdbadcdbbecfa02ea208
  • Make psql reject attempts to set special variables to invalid values. Previously, if the user set a special variable such as ECHO to an unrecognized value, psql would bleat but store the new value anyway, and then fall back to a default setting for the behavior controlled by the variable. This was agreed to be a not particularly good idea. With this patch, invalid values result in an error message and no change in state. (But this applies only to variables that affect psql's behavior; purely informational variables such as ENCODING can still be set to random values.) To do this, modify the API for psql's assign-hook functions so that they can return an OK/not OK result, and give them the responsibility for printing error messages when they reject a value. Adjust the APIs for ParseVariableBool and ParseVariableNum to support the new behavior conveniently. In passing, document the variable VERSION, which had somehow escaped that. And improve the quite-inadequate commenting in psql/variables.c. Daniel Vérité, reviewed by Rahila Syed, some further tweaking by me Discussion: https://postgr.es/m/7356e741-fa59-4146-a8eb-cf95fd6b21fb@mm http://git.postgresql.org/pg/commitdiff/511ae628f31b4e791cd5c7836e46cb84dcf145fd
  • Add a regression test script dedicated to exercising system views. Quite a few of our built-in system views were not exercised anywhere in the regression tests. This is perhaps not so exciting for the ones that are simple projections/joins of system catalogs, but for the ones that are wrappers for set-returning C functions, the omission translates directly to lack of test coverage for those functions. In many cases, the reason for the omission is that the view doesn't have much to do with any specific SQL feature, so there's no natural place to test it. To remedy that, invent a new script sysviews.sql that's dedicated to testing SRF-based views. Move a couple of tests that did fit this charter into the new script, and add simple "count(*)" based tests of other views within the charter. That's enough to ensure we at least exercise the main code path through the SRF, although it does little to prove that the output is sane. More could be done here, no doubt, and I hope someone will think about how we can test these views more thoroughly. But this is a starting point. Discussion: https://postgr.es/m/19359.1485723741@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/d002f16c6ec38f76d1ee97367ba6af3000d441d0
  • Invent pg_hba_file_rules view to show the content of pg_hba.conf. This view is designed along the same lines as pg_file_settings, to wit it shows what is currently in the file, not what the postmaster has loaded as the active settings. That allows it to be used to pre-vet edits before issuing SIGHUP. As with the earlier view, go out of our way to allow errors in the file to be reflected in the view, to assist that use-case. (We might at some point invent a view to show the current active settings, but this is not that patch; and it's not trivial to do.) Haribabu Kommi, reviewed by Ashutosh Bapat, Michael Paquier, Simon Riggs, and myself Discussion: https://postgr.es/m/CAJrrPGerH4jiwpcXT1-46QXUDmNp2QDrG9+-Tek_xC8APHShYw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/de16ab7238888b16825ad13f0bbe123632915e9b
  • Simplify some long-obsolete code in hba.c's next_token(). next_token() oddly set its buffer space consumption limit to one before the last char position in the buffer, not the last as you'd expect. The reason is there was once an ugly kluge to mark keywords by appending a newline to them, potentially requiring one more byte. Commit e5e2fc842 removed that kluge, but failed to notice that the length limit could be increased. Also, remove some vestigial handling of newline characters in the buffer. That was left over from when this function read the file directly using getc(). Commit 7f49a67f9 changed it to read from a buffer, from which tokenize_file had already removed the only possible occurrence of newline, but did not simplify this function in consequence. Also, ensure that we don't return with *lineptr set to someplace past the terminating '\0'; that would be catastrophic if a caller were to ask for another token from the same line. This is just latent since no callers actually do call again after a "false" return; but considering that it was actually costing us extra code to do it wrong, we might as well make it bulletproof. Noted while reviewing pg_hba_file_rules patch. http://git.postgresql.org/pg/commitdiff/1e5a5d03da14ba9b734c6a2a1a822dcd8af110eb
  • Make psql's \set display variables in alphabetical order. "\set" with no arguments displays all defined variables, but it does so in the order that they appear in variables.c's list, which previously was mostly creation order. That makes the list ugly and hard to find things in, and it exposes some psql implementation details to users. (For instance, ordinary variables will move to the bottom of the list if unset and set again, but variables that have hooks won't.) Fix that by keeping the list in alphabetical order at all times, which isn't much more complicated than breaking out of the insertion search loops once we reach an entry that should be after the one to be inserted. Discussion: https://postgr.es/m/31785.1485900786@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/c3e3844a92fe42109e4106314f7d229f784a7d0a
  • Improve psql's behavior for \set and \unset of its control variables. This commit improves on the results of commit 511ae628f in two ways: 1. It restores the historical behavior that "\set FOO" is interpreted as setting FOO to "on", if FOO is a boolean control variable. We already found one test script that was expecting that behavior, and the psql documentation certainly does nothing to discourage people from assuming that would work, since it often says just "if FOO is set" when describing the effects of a boolean variable. However, now this case will result in actually setting FOO to "on", not an empty string. 2. It arranges for an "\unset" of a control variable to set the value back to its default value, rather than becoming apparently undefined. The control variables are also initialized that way at psql startup. In combination, these things guarantee that a control variable always has a displayable value that reflects what psql is actually doing. That is a pretty substantial usability improvement. The implementation involves adding a second type of variable hook function that is able to replace a proposed new value (including NULL) with another one. We could alternatively have complicated the API of the assign hook, but this way seems better since many variables can share the same substitution hook function. Also document the actual behavior of these variables more fully, including covering assorted behaviors that were there before but never documented. This patch also includes some minor cleanup that should have been in 511ae628f but was missed. Patch by me, but it owes a lot to discussions with Daniel Vérité. Discussion: https://postgr.es/m/9572.1485821620@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/86322dc7e013b4062393dcbb74043db003e23ec5
  • Provide CatalogTupleDelete() as a wrapper around simple_heap_delete(). This extends the work done in commit 2f5c9d9c9 to provide a more nearly complete abstraction layer hiding the details of index updating for catalog changes. That commit only invented abstractions for catalog inserts and updates, leaving nearby code for catalog deletes still calling the heap-level routines directly. That seems rather ugly from here, and it does little to help if we ever want to shift to a storage system in which indexing work is needed at delete time. Hence, create a wrapper function CatalogTupleDelete(), and replace calls of simple_heap_delete() on catalog tuples with it. There are now very few direct calls of [simple_]heap_delete remaining in the tree. Discussion: https://postgr.es/m/462.1485902736@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/ab02896510e26e46b830c87eef2c03dd3c52c09e
  • Fix CatalogTupleInsert/Update abstraction for case of shared indstate. Add CatalogTupleInsertWithInfo and CatalogTupleUpdateWithInfo to let callers use the CatalogTupleXXX abstraction layer even in cases where we want to share the results of CatalogOpenIndexes across multiple inserts/updates for efficiency. This finishes the job begun in commit 2f5c9d9c9, by allowing some remaining simple_heap_insert/update calls to be replaced. The abstraction layer is now complete enough that we don't have to export CatalogIndexInsert at all anymore. Also, this fixes several places in which 2f5c9d9c9 introduced performance regressions by using retail CatalogTupleInsert or CatalogTupleUpdate even though the previous coding had been able to amortize CatalogOpenIndexes work across multiple tuples. A possible future improvement is to arrange for the indexing.c functions to cache the CatalogIndexState somewhere, maybe in the relcache, in which case we could get rid of CatalogTupleInsertWithInfo and CatalogTupleUpdateWithInfo again. But that's a task for another day. Discussion: https://postgr.es/m/27502.1485981379@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/aedd554f84bb3cedb32e6e2a954a70537a4da6b9
  • Fix mishandling of tSRFs at different nesting levels. Given a targetlist like "srf(x), f(srf(x))", split_pathtarget_at_srfs() decided that it needed two levels of ProjectSet nodes, failing to notice that the two SRF calls are textually equal(). Because of that, setrefs.c would convert the upper ProjectSet's tlist to "Var1, f(Var1)" (where Var1 represents a reference to the srf(x) output of the lower ProjectSet). This triggered an assertion in nodeProjectSet.c complaining that it found no SRFs to evaluate, as reported by Erik Rijkers. What we want in such a case is to evaluate srf(x) only once and use a plain Result node to compute "Var1, f(Var1)"; that gives results similar to what previous versions produced, whereas allowing srf(x) to be evaluated again in an upper ProjectSet would square the number of rows emitted. Furthermore, even if the SRF calls aren't textually identical, we want them to be evaluated in lockstep, because that's what happened in the old implementation. But split_pathtarget_at_srfs() got this completely wrong, using two levels of ProjectSet for a case like "srf(x), f(srf(y))". Hence, rewrite split_pathtarget_at_srfs() from the ground up so that it groups SRFs according to the depth of nesting of SRFs in their arguments. This is pretty much how we envisioned that working originally, but I blew it when it came to implementation. In passing, optimize the case of target == input_target, which I noticed is not only possible but quite common. Discussion: https://postgr.es/m/dcbd2853c05d22088766553d60dc78c6@xs4all.nl http://git.postgresql.org/pg/commitdiff/c82d4e658eff287846fe6243966d110ff4ae9025
  • Fix placement of initPlans when forcibly materializing a subplan. If we forcibly place a Material node atop a finished subplan, we need to move any initPlans attached to the subplan up to the Material node, in order to keep SS_finalize_plan() happy. I'd figured this out in commit 7b67a0a49 for the case of materializing a cursor plan, but out of an abundance of caution, I put the initPlan movement hack at the call site for that case, rather than inside materialize_finished_plan(). That was the wrong thing, because it turns out to also be necessary for the only other caller of materialize_finished_plan(), ie subselect.c. We lacked any test cases that exposed the mistake, but bug#14524 from Wei Congrui shows that it's possible to get an initPlan reference into the top tlist in that case too, and then SS_finalize_plan() complains. Hence, move the hack into materialize_finished_plan(). In HEAD, also relocate some recently-added tests in subselect.sql, which I'd unthinkingly dropped into the middle of a sequence of related tests. Report: https://postgr.es/m/20170202060020.1400.89021@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/555494d1bc119173bbf712940da823303644d4de
  • Avoid improbable null pointer dereference in pgpassfileWarning(). Coverity complained that we might pass a null pointer to strcmp() if PQresultErrorField were to return NULL. That shouldn't be possible, since the server is supposed to always provide some SQLSTATE or other in an error message. But we usually defend against such hazards, and it only takes a little more code to do so here. There's no good reason to think this is a live bug, so no back-patch. http://git.postgresql.org/pg/commitdiff/8ac0365c22f20dcd2a6b6752b95b665ada83e858
  • Clean up psql's behavior for a few more control variables. Modify FETCH_COUNT to always have a defined value, like other control variables, mainly so it will always appear in "\set" output. Add hooks to force HISTSIZE to be defined and require it to have an integer value. (I don't see any point in allowing it to be set to non-integral values.) Add hooks to force IGNOREEOF to be defined and require it to have an integer value. Unlike the other cases, here we're trying to be bug-compatible with a rather bogus externally-defined behavior, so I think we need to continue to allow "\set IGNOREEOF whatever". Fix it so that the substitution hook silently replace non-numeric values with "10", so that the stored value always reflects what we're really doing. Add a dummy assign hook for HISTFILE, just so it's always in variables.c's list. We can't require it to be defined always, because that would break the interaction with the PSQL_HISTORY environment variable, so there isn't any change in visible behavior here. Remove tab-complete.c's private list of known variable names, since that's really a maintenance nuisance. Given the preceding changes, there are no control variables it won't show anyway. This does mean that if for some reason you've unset one of the status variables (DBNAME, HOST, etc), that variable would not appear in tab completion for \set. But I think that's fine, for at least two reasons: we shouldn't be encouraging people to use those variables as regular variables, and if someone does do so anyway, why shouldn't it act just like a regular variable? Remove ugly and no-longer-used-anywhere GetVariableNum(). In general, future additions of integer-valued control variables should follow the paradigm of adding an assign hook using ParseVariableNum(), so there's no reason to expect we'd need this again later. Discussion: https://postgr.es/m/17516.1485973973@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/fd6cd698031d3dbeccafde49a8b118cb840312aa
  • Fix a bunch more portability bugs in commit 08bf6e529. It seems like somebody used a dartboard while choosing integer widths for the various values taken and returned by these functions ... and then threw a fresh set of darts while writing the SQL declarations. This patch brings the C code into line with what the SQL declarations say, which is enough to make it not dump core on the particular 32-bit machine I'm testing on. But I think we could do with another round of looking at what the datum widths *should* be. For instance, it's not all that sensible that hash_bitmap_info decided to use int64 to represent a BlockNumber input when get_raw_page doesn't do it that way. There's also a remaining problem that the expected outputs from the test script are platform-dependent, but I'll leave that issue for somebody else. Per buildfarm. http://git.postgresql.org/pg/commitdiff/c6eeb67dcc3b3f7aa94335e18a1075863d808a7c
  • In pageinspect/hashfuncs.c, avoid crashes on alignment-picky machines. On machines with MAXALIGN = 8, the payload of a bytea is not maxaligned, since it will start 4 bytes into a palloc'd value. On alignment-picky hardware, this will cause failures in accesses to 8-byte-wide values within the page. We already encountered this problem when we introduced GIN index inspection functions, and fixed it in commit 84ad68d64. Make use of the same function for hash indexes. A small difficulty is that up to now contrib/pageinspect has not shared any functions at all across files. To support that, introduce a common header file "pageinspect.h" for the module. Also, move get_page_from_raw() out of ginfuncs.c, where it didn't especially belong, and put it in rawpage.c which seems a more natural home. Per buildfarm. Discussion: https://postgr.es/m/17311.1486134714@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/14e9b18fed289e483ed28daec60fdab95da9cc48
  • First-draft release notes for 9.6.2. As usual, the release notes for other branches will be made by cutting these down, but put them up for community review first. http://git.postgresql.org/pg/commitdiff/9863017b87f3592ff663d03fc663a4f1f8fdb8b2

Peter Eisentraut pushed:

Ãlvaro Herrera pushed:

  • Tweak catalog indexing abstraction for upcoming WARM. Split the existing CatalogUpdateIndexes into two different routines, CatalogTupleInsert and CatalogTupleUpdate, which do both the heap insert/update plus the index update. This removes over 300 lines of boilerplate code all over src/backend/catalog/ and src/backend/commands. The resulting code is much more pleasing to the eye. Also, by encapsulating what happens in detail during an UPDATE, this facilitates the upcoming WARM patch, which is going to add a few more lines to the update case making the boilerplate even more boring. The original CatalogUpdateIndexes is removed; there was only one use left, and since it's just three lines, we can as well expand it in place there. We could keep it, but WARM is going to break all the UPDATE out-of-core callsites anyway, so there seems to be no benefit in doing so. Author: Pavan Deolasee Discussion: https://www.postgr.es/m/CABOikdOcFYSZ4vA2gYfs=M2cdXzXX4qGHeEiW3fu9PCfkHLa2A@mail.gmail.com http://git.postgresql.org/pg/commitdiff/2f5c9d9c9cec436e55847ec580606d7e88067df6

Robert Haas pushed:

Andrew Dunstan pushed:

  • Don't count background workers against a user's connection limit. Doing so doesn't seem to be within the purpose of the per user connection limits, and has particularly unfortunate effects in conjunction with parallel queries. Backpatch to 9.6 where parallel queries were introduced. David Rowley, reviewed by Robert Haas and Albe Laurenz. http://git.postgresql.org/pg/commitdiff/f1169ab501ce90e035a7c6489013a1d4c250ac92

Noah Misch pushed:

  • Code review for avoidance of direct cross-module links. Remove $(pkglibdir) from $(rpathdir), since commits d51924be886c2a05e691fa05b16cb6b30ab8370f and eda04886c1e048d695728206504ab4198462168e removed direct linkage to objects stored there. Users are unlikely to notice the difference. Accompany every $(python_libspec) with $(python_additional_libs); this doesn't fix a demonstrated bug, but it might do so on rare Python configurations. With these changes, AIX ceases to be a special case. http://git.postgresql.org/pg/commitdiff/acd73ad1a17f7aed6a914dd9c331f1043d89988d

Fujii Masao pushed:

Correctifs en attente

Takayuki Tsunakawa sent in another revision of a patch to support huge pages on Windows.

Nikita Glukhov sent in three revisions of a patch to fix a bug in SP-GiST box_ops.

Kyotaro HORIGUCHI sent in two more revisions of a patch to implement character conversion with radix trees.

Alexander Korotkov sent in another revision of a patch to cacheline align PGXACT.

Nikita Glukhov sent in a patch to implement kNN searches for SP-GiST.

Michaël Paquier sent in another revision of a patch to allow interrupts on waiting standby.

Rushabh Lathia sent in a patch to extend the existing wait event infrastructure to implement the wait events for the disk I/O.

Etsuro Fujita sent in another revision of a patch to push down more full joins to the PostgreSQL FDW.

Michaël Paquier sent in two more revisions of a patch to refactor replication commands using printsimple.

David Rowley sent in two more revisions of a patch to for improve performance for outer joins where outer side is unique.

Daisuke Higuchi and Ashutosh Bapat traded patches to fix a bug where PQgetResult is not called until PQgetResult has returned a null pointer.

Heikki Linnakangas sent in two more revisions of a patch to fix an issue which produced a deadlock in XLogInsert on AIX.

Kyotaro HORIGUCHI sent in another revision of a patch to implement asynchronous execution.

Tomas Vondra sent in another revision of a patch to implement multivariate statistics.

Amit Kapila sent in two more revisions of a patch to implement parallel index scans.

Claudio Freire sent in two more revisions of a patch to enable VACUUM to use more than 1GB of work_mem.

Pavan Deolasee, Ãlvaro Herrera, and Tom Lane traded patches to fix an issue where CREATE INDEX CONCURRENTLY could cause index corruption.

Amit Kapila sent in another revision of a patch to parallelize queries containing initplans.

Pavel Stěhule sent in another revision of a patch to implement xmltable().

Michaël Paquier sent in a patch to forbid newline and carriage return characters in database and role names, and ensure clean-up of data directory even with restricted path applied.

Haribabu Kommi sent in another revision of a patch to add a macaddr 64 bit (EUI-64) datatype.

Peter Eisentraut sent in another revision of a patch to implement a set of SEQUENCE data types.

Pavan Deolasee sent in three more revisions of a patch to implement WARM.

Kyotaro HORIGUCHI sent in two more revisions of a patch to fix a bug in physical replication slots.

Kyotaro HORIGUCHI sent in two more revisions of a patch to enable logical replication between databases with different encodings.

Dilip Kumar sent in two more revisions of a patch to implement parallel bitmap heap scan.

Pavel Stěhule sent in another revision of a patch to enable PL/pgsql to force a custom or generic plan.

Anastasia Lubennikova sent in a patch to allow casting from the appropriate jsonb objects to numeric, int, float, bool.

Dagfinn Ilmari Mannsåker sent in two revisions of a patch to add tab completion for DEALLOCATE.

Rushabh Lathia sent in two more revisions of a patch to implement Gather Merge.

Nikhil Sontakke sent in two more revisions of a patch to speed up two-phase transactions.

David Rowley sent in a patch to fix a compiler warning in dbd69118.

Michaël Paquier sent in two revisions of a patch to provide list of subscriptions and publications in psql's completion.

Corey Huinker sent in three more revisions of a patch to add \if and friends to psql.

Peter Moser sent in another revision of a patch to add temporal query primitives.

Alexey Bashtanov sent in a patch to optimize the query used to produce information_schema.constraint_column_usage.

Simon Riggs sent in a patch to introduce the concept of a "superowner".

Andres Freund sent in a patch to implement a basic lockless single producer, multiple consumer ringbuffer, and a rewrite of the bgwriter to use same.

Kyotaro HORIGUCHI sent in another revision of a patch to refactor tab completion in psql and use same to add some tab completion features.

Amit Langote sent in a patch to improve the documentation about partitioned tables.

Andreas Karlsson sent in a patch to add psql tab completion for \help DROP|ALTER.

Alexander Korotkov sent in a patch to implement low-level locking on PowerPC in assembly.

Boris Muratshin sent in a patch to implement 3D Z-curve spatial indexes.

Fabien COELHO sent in another revision of a patch to add more operators and functions to pgbench.

Alexander Korotkov sent in a patch to re-add strategy to PinBuffer.

Masahiko Sawada sent in a patch to fix a typo in a variable name in launcher.c.

Noah Misch sent in a patch to ignore tablespace ACLs when ignoring schema ACLs.

Brandur Leach sent in another revision of a patch to implement SortSupport for macaddr data type.

par N Bougain le mercredi 8 février 2017 à 23h34

mardi 7 février 2017

Nicolas Gollet

Backends générant des fichiers temporaires

Dans certain cas il est peut être utile d'obtenir en temps réel la liste des backends générant des fichiers temporaire sur disque.

Lors de certaines opérations (trie, hachages...), PostgreSQL écrit dans des fichiers temporaires. Ces fichiers sont placés dans le sous-répertoire pgsql_tmp et sont nommés de la façon suivante :
pgsql_tmpXXXX.YXXXX correspond au PID du processus qui a créé le fichier et Y au numéro de fichier créé. Par exemple voici deux fichiers temporaires : pgsql_tmp90630.8 et pgsql_tmp90630.9. Lorsque vous utilisé des tablespaces ces fichiers temporaires sont stockés dans l'emplacement disque définit par ceux-ci ceux qui rajoute une complexité supplémentaire pour récupéré ces fichiers.

Une fois le PID identifié, vous pouvez obtenir les informations depuis la vue pg_stat_activity afin d'obtenir les informations sur le backend générant des fichiers temporaires :
select * from pg_stat_activity where pid = <PID>;

Afin de simplifier la récupération de ces informations vous pouvez obtenir ces informations de façon automatique en utilisant la requête SQL ci-dessous :

Cette requête adaptée du projet pgstats permet d'obtenir l'ensemble des requêtes en cours d’exécution générant des fichiers temporaires tout en prenant en compte les éventuelles tablespace.

Vous pouvez aussi utiliser le script bash psql_show_tempfiles.bash présent sur mon Gitub.

par Nicolas GOLLET le mardi 7 février 2017 à 15h44

dimanche 5 février 2017

Damien Clochard

PostgreSQL Temboard : Guide de Démarrage

Le prochain défi pour la communauté PostgreSQL est de produire des outils graphiques riches et bien conçus. C’est un domain dans lequel nous avons une grande marge de progression et Temboard est un des projets (parmi d’autres) qui a l’ambition de devenir un outil graphique pour PostgreSQL.

Temboard : PostgreSQL Remote Control

Puisque que nous parlons d’un outil graphique, allons directement jetez un coup d’oeuil ! Voici 3 étapes pour déployer un environement de test composé de 3 instances Postgres et un serveur central Temboard.

1- Installer docker et Installer docker-compose

2- Récuperer fichier docker compose pré-configuré et le lancer

wget https://raw.githubusercontent.com/dalibo/docker/master/temboard/docker-compose.yml
docker-compose up

3- Aller sur https://127.0.0.1:8888

  • Connectez-vous sur le serveur Temboard avec les codes admin / admin
  • Connectez-vous sur chaque instance Postgres avec alice / alice

Vous devriez voir apparaitre quelque chose comme ça :

Temboard PostgreSQL Dashboard

C’est tout pour aujourd’hui ! Je ferai un autre article prochainement pour présenter les fonctionnalités de Temboard notamment le tableau de bord, la versionnement des fichiers postgresql.conf, comment tuer une transaction, etc.

En attendant si vous prenez le temps de tester cet outil, envoyez nous un message à contact@dalibo.com et dites-nous ce que vous en pensez, quelle fonctionnalités vous manquent !

Temboard est encore un projet très jeune et nous avons prévu d’ajouter plus de plugins dans les mois à venir !

par Damien Clochard le dimanche 5 février 2017 à 10h17

jeudi 2 février 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 29 janvier 2017

Vous pouvez bénéficier de 50% de remise pour le SCALE 15x avec le code PGDAY : https://www.socallinuxexpo.org/scale/15x

[ndt: Meetup à Lyon le 21 février :https://www.meetup.com/fr-FR/PostgreSQL-Lyon-User-Group/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170129225258.GC17445@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Peter Eisentraut pushed:

Tom Lane pushed:

  • Volatile-ize some plperl variables that must survive into PG_CATCH blocks. This appears to be necessary to fix a failure seen on buildfarm member sittella. It shouldn't be necessary according to the letter of the C standard, because we don't change the values of these variables within the PG_TRY blocks; but somehow gcc 4.7.2 is dropping the ball. Discussion: https://postgr.es/m/17555.1485179975@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/c0ef456b9714215cae0ce3561c7c83629a7301cf
  • Fix example plan in optimizer/README. Joining three tables only takes two join nodes. I think when I (tgl) wrote this, I was envisioning possible additional joins; but since the example doesn't show any fourth table, it's just confusing to write a third join node. Etsuro Fujita Discussion: https://postgr.es/m/e6cfbaa3-af02-1abc-c25e-8fa5c6bc4e21@lab.ntt.co.jp http://git.postgresql.org/pg/commitdiff/3c821466abcdb8120ab0dfbce02f3bbae3b62025
  • Allow password file name to be specified as a libpq connection parameter. Formerly an alternate password file could only be selected via the environment variable PGPASSFILE; now it can also be selected via a new connection parameter "passfile", corresponding to the conventions for most other connection parameters. There was some concern about this creating a security weakness, but it was agreed that that argument was pretty thin, and there are clear use-cases for handling password files this way. Julian Markwort, reviewed by Fabien Coelho, some adjustments by me Discussion: https://postgr.es/m/a4b4f4f1-7b58-a0e8-5268-5f7db8e8ccaa@uni-muenster.de http://git.postgresql.org/pg/commitdiff/ba005f193d88a8404e81db3df223cf689d64d75e
  • Use non-conflicting table names in new regression test case. Commit 587cda35c added a test to updatable_views.sql that created tables named the same as tables used by the concurrent inherit.sql script. Unsurprisingly, this results in random failures. Pick different names. Per buildfarm. http://git.postgresql.org/pg/commitdiff/7fa7bf18e493e130147e62cf7dc33010f164126c
  • Improve speed of contrib/postgres_fdw regression tests. Commit 7012b132d added some tests that consumed an excessive amount of time, more than tripling the time needed for "make installcheck" for this module. Add filter conditions to reduce the number of rows scanned, bringing the runtime down to within hailing distance of what it was before. Jeevan Chalke and Ashutosh Bapat, per a gripe from me Discussion: https://postgr.es/m/16565.1478104765@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/aa7f593b1ffa9717bd5570174944c06c482d1c1f
  • Change unknown-type literals to type text in SELECT and RETURNING lists. Previously, we left such literals alone if the query or subquery had no properties forcing a type decision to be made (such as an ORDER BY or DISTINCT clause using that output column). This meant that "unknown" could be an exposed output column type, which has never been a great idea because it could result in strange failures later on. For example, an outer query that tried to do any operations on an unknown-type subquery output would generally fail with some weird error like "failed to find conversion function from unknown to text" or "could not determine which collation to use for string comparison". Also, if the case occurred in a CREATE VIEW's query then the view would have an unknown-type column, causing similar failures in queries trying to use the view. To fix, at the tail end of parse analysis of a query, forcibly convert any remaining "unknown" literals in its SELECT or RETURNING list to type text. However, provide a switch to suppress that, and use it in the cases of SELECT inside a set operation or INSERT command. In those cases we already had type resolution rules that make use of context information from outside the subquery proper, and we don't want to change that behavior. Also, change creation of an unknown-type column in a relation from a warning to a hard error. The error should be unreachable now in CREATE VIEW or CREATE MATVIEW, but it's still possible to explicitly say "unknown" in CREATE TABLE or CREATE (composite) TYPE. We want to forbid that because it's nothing but a foot-gun. This change creates a pg_upgrade failure case: a matview that contains an unknown-type column can't be pg_upgraded, because reparsing the matview's defining query will now decide that the column is of type text, which doesn't match the cstring-like storage that the old materialized column would actually have. Add a checking pass to detect that. While at it, we can detect tables or composite types that would fail, essentially for free. Those would fail safely anyway later on, but we might as well fail earlier. This patch is by me, but it owes something to previous investigations by Rahila Syed. Also thanks to Ashutosh Bapat and Michael Paquier for review. Discussion: https://postgr.es/m/CAH2L28uwwbL9HUM-WR=hromW1Cvamkn7O-g8fPY2m=_7muJ0oA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/1e7c4bb0049732ece651d993d03bb6772e5d281a
  • Make UNKNOWN into an actual pseudo-type. Previously, type "unknown" was labeled as a base type in pg_type, which perhaps had some sense to it because you were allowed to create tables with unknown-type columns. But now that we don't allow that, it makes more sense to label it a pseudo-type. This has the additional effects of forbidding use of "unknown" as a domain base type, cast source or target type, PL function argument or result type, or plpgsql local variable type; all of which seem like good holes to plug. Discussion: https://postgr.es/m/CAH2L28uwwbL9HUM-WR=hromW1Cvamkn7O-g8fPY2m=_7muJ0oA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/d8d32d9a56a3cecfb14e8f47ebd50b780edffe60
  • Remove vestigial resolveUnknown arguments from transformSortClause etc. There's really no situation where we don't want these unknown-to-text conversions to happen. The alternative is failure anyway, and the one caller that was passing "false" did so only because it expected the case could not arise. Might as well simplify the code. Discussion: https://postgr.es/m/CAH2L28uwwbL9HUM-WR=hromW1Cvamkn7O-g8fPY2m=_7muJ0oA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/aebeb4790c750dc808c1c5afb3cb435116244e36
  • Introduce convenience macros to hide JsonbContainer header accesses better. This improves readability a bit and may make future improvements easier. In passing, make sure that the JB_ROOT_IS_XXX macros deliver boolean (0/1) results; the previous coding was a bug hazard, though no actual bugs are known. Nikita Glukhov, extended a bit by me Discussion: https://postgr.es/m/9e21a39c-c1d7-b9b5-44a0-c5345a5029f6@postgrespro.ru http://git.postgresql.org/pg/commitdiff/f7c62462402972b13d10e43f104ca0c0fecb6d08
  • Ensure that a tsquery like '!foo' matches empty tsvectors. !foo means "the tsvector does not contain foo", and therefore it should match an empty tsvector. ts_match_vq() overenthusiastically supposed that an empty tsvector could never match any query, so it forcibly returned FALSE, the wrong answer. Remove the premature optimization. Our behavior on this point was inconsistent, because while seqscans and GIST index searches both failed to match empty tsvectors, GIN index searches would find them, since GIN scans don't rely on ts_match_vq(). That makes this certainly a bug, not a debatable definition disagreement, so back-patch to all supported branches. Report and diagnosis by Tom Dunstan (bug #14515); added test cases by me. Discussion: https://postgr.es/m/20170126025524.1434.97828@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/9d4ca01314ba0c571139c5560a40ef764ad0feee
  • Use castNode() in a bunch of statement-list-related code. When I wrote commit ab1f0c822, I really missed the castNode() macro that Peter E. had proposed shortly before. This back-fills the uses I would have put it to. It's probably not all that significant, but there are more assertions here than there were before, and conceivably they will help catch any bugs associated with those representation changes. I left behind a number of usages like "(Query *) copyObject(query_var)". Those could have been converted as well, but Peter has proposed another notational improvement that would handle copyObject cases automatically, so I let that be for now. http://git.postgresql.org/pg/commitdiff/7afd56c3c6d8360a5bfdfb2de30038b239fd756b
  • Orthography fixes for new castNode() macro. Clean up hastily-composed comment. Normalize whitespace. Erik Rijkers and myself http://git.postgresql.org/pg/commitdiff/fefb86b14776321ac153836398eadde867ff31af
  • Improve comments about ProcessUtility's queryString parameter. Per discussion with Craig Ringer. http://git.postgresql.org/pg/commitdiff/fde5c037925b01b937923606c39460d94965672e
  • Restructure hba.c to replace 3 parallel lists with single list of structs. tokenize_file() now returns a single list of TokenizedLine structs, carrying the same information as before. We were otherwise going to grow a fourth list to deal with error messages, and that was getting a bit silly. Haribabu Kommi, revised a bit by me Discussion: https://postgr.es/m/CAJrrPGfbgbKsjYp=bgZXhMcgxoaGSoBb9fyjrDoOW_YymXv1Kw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/350cb921ae2ced8674e098d0645f2493e5757ad1

Ãlvaro Herrera pushed:

Tatsuo Ishii pushed:

Fujii Masao pushed:

Robert Haas pushed:

Simon Riggs pushed:

Andres Freund pushed:

Correctifs en attente

Michaël Paquier sent in a patch to enable replication connections by default in pg_hba.conf.

Michaël Paquier sent in another revision of a patch to ensure that pg_dump and pg_dumpall sync their output to disk by default.

Michaël Paquier sent in a patch to ensure that launching replication workers is a NOOP when wal_level < logical, and to set the default number of such workers to 0.

Craig Ringer sent in three more revisions of a patch to fix a race between clog truncation and lookup, and introduce txid_status(bigint) to get status of an xact.

Petr Jelínek sent in two more revisions of a patch to use the asynchronous connect API in libpqwalreceiver, close the replication connection when slot creation gets canceled, ensure that stringinfo buffers in walsender are always initialized, fix after trigger execution in logical replication, and add RENAME support for PUBLICATIONs and SUBSCRIPTIONs.

Amit Kapila sent in another revision of a patch to implement parallel index scans.

Dilip Kumar sent in three more revisions of a patch to implement parallel bitmap heap scans.

Beena Emerson sent in another revision of a patch to enable increasing the default WAL segment size.

Corey Huinker sent in three more revisions of a patch to add \if and friends to psql.

Etsuro Fujita sent in another revision of a patch to fix a bug in the PostgreSQL FDW.

Ivan Kartyshov sent in another revision of a patch to make async slave to wait for lsn to be replayed.

Dagfinn Ilmari Mannsåker sent in another revision of a patch to add GUCs for predicate lock promotion thresholds.

Craig Ringer sent in another revision of a patch to implement logical decoding on standby.

Nico Williams sent in a patch to implement an expanded version of materialized views and a contrib extension.

Nico Williams sent in a patch to implement pqasyncnotifier.c, a shell command client for LISTEN.

Amit Langote sent in a patch to add relkind checks to certain contrib modules.

Kyotaro HORIGUCHI sent in another revision of a patch to clean up the negative cache of pg_statistic when dropping a relation and of pg_class when dropping a schema.

Amit Kapila sent in another revision of a patch to parallelize queries containing subplans.

Ashutosh Sharma sent in two more revisions of a patch to add pgstathashindex() to pgstattuple extension.

Jim Nasby sent in another revision of a patch to add faster methods for getting SPI results.

Claudio Freire sent in two more revisions of a patch to allow usage of more than 1GB of work mem in VACUUM.

Pavel Stěhule sent in two revisions of a patch to enable forcing a custom or generic plan in PL/pgsql.

Stas Kelvich and Nikhil Sontakke traded patches to speed up two-phase transactions.

Haribabu Kommi sent in three more revisions of a patch to implement a 64-bit (EUI-64) macaddr data type.

Haribabu Kommi and Tom Lane traded patches to implement a pg_hba_file_settings view.

Peter Eisentraut sent in another revision of a patch to add ICU support.

Beena Emerson sent in a patch to add tab completion to ALTER in psql.

Daniel Vérité sent in another revision of a patch to improve psql hooks for variables.

Nikita Glukhov sent in another revision of a patch to make a recursive version of json_populate_record().

Ãlvaro Herrera sent in another revision of a patch to implement xmltable().

Fabien COELHO sent in two more revisions of a patch to add more functions and operators to pgbench.

Pavan Deolasee sent in another revision of a patch to implement WARM.

Michaël Paquier sent in three revisions of a patch to refactor the replication commands output.

Vladimir Rusinov sent in a patch to rename the sql-callable functions with xlog to have wal instead.

Etsuro Fujita sent in another revision of a patch to push down more UPDATEs/DELETEs in postgres_fdw.

Ashutosh Bapat sent in another revision of a patch to speed up aggregate pushdown tests.

Julian Markwort sent in a patch to extended the functionality of pg_stat_statements so it can track worst and best case execution plans.

Fabien COELHO sent in a patch to fix an infelicity between pgbench's --connect and --rate options.

Kyotaro HORIGUCHI sent in another revision of a patch to use a radix tree for character conversion.

Peter Eisentraut sent in two more revisions of a patch to add test coverage for sequences.

David Fetter and Corey Huinker traded patches to add copy_srf(), a set-returning function corresponding to COPY IN.

Masahiko Sawada sent in two more revisions of a patch to support 2PC in FDWs.

Michaël Paquier sent in two revisions of a patch to remove race conditions between the checkpointer and the init fork creations by making index init forks go through the shared buffers instead of having their empty() routines handle the flush of the page created.

Robert Haas sent in a patch to remove some hard-coded superuser checks.

Robert Haas sent in another revision of a patch to rename things *xlog* to the corresponding *wal*.

Mithun Cy sent in two more revisions of a patch to cache hash index meta pages.

David Rowley sent in three more revisions of a patch to improve performance where for joins where the outer side is unique.

Venkata B Nagothi sent in a patch to generate an error by aborting the recovery process instead of starting up the cluster if the intended recovery target point is not reached, and give an option to DBA to resume the recovery process from exactly where it stopped.

Ashutosh Sharma sent in another revision of a patch to add microvacuum support for hash indexes.

Noah Misch sent in another revision of a patch to remove link-time cross-module refs in contrib.

Christoph Berg sent in a patch to use \G to use expanded output for a query or current query buffer.

Robert Haas and Ãlvaro Herrera traded patches to add hash index support to to the pageinspect contrib extension.

Thomas Munro sent in another revision of a patch to implement parallel shared hash.

David Rowley sent in another revision of a patch to fix an infelicity between CONNECTION LIMIT and Parallel Query.

Tom Lane sent in a patch to create a separate test file for exercising system views.

par N Bougain le jeudi 2 février 2017 à 00h53

Nouvelles hebdomadaires de PostgreSQL - 22 janvier 2017

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170122232424.GA6120@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Fujii Masao pushed:

  • Fix typos in comments. Masahiko Sawada http://git.postgresql.org/pg/commitdiff/8fa6019b405f9d39539a77c6f5f11fe640ddf955
  • Fix an assertion failure related to an exclusive backup. Previously multiple sessions could execute pg_start_backup() and pg_stop_backup() to start and stop an exclusive backup at the same time. This could trigger the assertion failure of "FailedAssertion("!(XLogCtl->Insert.exclusiveBackup)". This happend because, even while pg_start_backup() was starting an exclusive backup, other session could run pg_stop_backup() concurrently and mark the backup as not-in-progress unconditionally. This patch introduces ExclusiveBackupState indicating the state of an exclusive backup. This state is used to ensure that there is only one session running pg_start_backup() or pg_stop_backup() at the same time, to avoid the assertion failure. Back-patch to all supported versions. Author: Michael Paquier Reviewed-By: Kyotaro Horiguchi and me Reported-By: Andreas Seltenreich Discussion: <87mvktojme.fsf@credativ.de> http://git.postgresql.org/pg/commitdiff/974ece58bbb3c0ef185a9d44b1cedae51cd56b04
  • Add description of temporary column into pg_replication_slots doc. Ayumi Ishii http://git.postgresql.org/pg/commitdiff/954737095061e5b5f1d87fb8cc43f7f8afff64c6

Magnus Hagander pushed:

Tom Lane pushed:

  • Fix NULL pointer dereference in tuplesort.c. Oversight in commit e94568ecc. This could cause a crash when an external datum tuplesort of a pass-by-value type required multiple passes. Per report from Mithun Cy. Peter Geoghegan Discussion: https://postgr.es/m/CAD__OujuhfWFULGFSt1fyHqUb8N-XafjJhudwt88V0Qs2o84qg@mail.gmail.com http://git.postgresql.org/pg/commitdiff/4e46c97fde42fa8ca57d29b9b47f2ebd11ab8105
  • Fix check_srf_call_placement() to handle VALUES cases correctly. INSERT ... VALUES with a single VALUES row is implemented quite differently from the general VALUES case. A user-visible implication of that is that we accept SRFs in the single-row case, but not in the multi-row case. That's a historical artifact no doubt, but in view of the lack of field complaints, I'm not excited about fixing it right now. However, check_srf_call_placement() needs to know about this, first because it should throw an error in the unsupported case, and second because it should set p_hasTargetSRFs in the single-row case (because we treat that like a SELECT tlist). That's an oversight in commit a4c35ea1c. To fix, split EXPR_KIND_VALUES into two values. So far as I can see, this is the only place where we need to distinguish the two cases at present; but there might be more later. Patch by me, per report from Andres Freund. Discussion: https://postgr.es/m/20170116081548.zg63zltblwimpfgp@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/d43a619c60405ecda275ca9e3ac9ead242e20ecb
  • Improve RLS planning by marking individual quals with security levels. In an RLS query, we must ensure that security filter quals are evaluated before ordinary query quals, in case the latter contain "leaky" functions that could expose the contents of sensitive rows. The original implementation of RLS planning ensured this by pushing the scan of a secured table into a sub-query that it marked as a security-barrier view. Unfortunately this results in very inefficient plans in many cases, because the sub-query cannot be flattened and gets planned independently of the rest of the query. To fix, drop the use of sub-queries to enforce RLS qual order, and instead mark each qual (RestrictInfo) with a security_level field establishing its priority for evaluation. Quals must be evaluated in security_level order, except that "leakproof" quals can be allowed to go ahead of quals of lower security_level, if it's helpful to do so. This has to be enforced within the ordering of any one list of quals to be evaluated at a table scan node, and we also have to ensure that quals are not chosen for early evaluation (i.e., use as an index qual or TID scan qual) if they're not allowed to go ahead of other quals at the scan node. This is sufficient to fix the problem for RLS quals, since we only support RLS policies on simple tables and thus RLS quals will always exist at the table scan level only. Eventually these qual ordering rules should be enforced for join quals as well, which would permit improving planning for explicit security-barrier views; but that's a task for another patch. Note that FDWs would need to be aware of these rules --- and not, for example, send an insecure qual for remote execution --- but since we do not yet allow RLS policies on foreign tables, the case doesn't arise. This will need to be addressed before we can allow such policies. Patch by me, reviewed by Stephen Frost and Dean Rasheed. Discussion: https://postgr.es/m/8185.1477432701@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/215b43cdc8d6b4a1700886a39df1ee735cb0274d
  • Avoid conflicts with collation aliases generated by stripping. This resulted in failures depending on the order of "locale -a" output. The original coding in initdb sorted the results, but that should be unnecessary as long as "locale -a" doesn't print duplicate names. The original entries will then all be non-dups, and while we might generate duplicate aliases by stripping, they should be for different encodings and thus not conflict. Even if the latter assumption fails somehow, it won't be fatal because we're using if_not_exists mode for the aliases. Discussion: https://postgr.es/m/26116.1484751196%40sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/0333a7340054c3356940759b1ab2391eed572171
  • Disable transforms that replaced AT TIME ZONE with RelabelType. These resulted in wrong answers if the relabeled argument could be matched to an index column, as shown in bug #14504 from Evgeniy Kozlov. We might be able to resurrect these optimizations by adjusting the planner's treatment of RelabelType, or by adjusting btree's rules for selecting comparison functions, but either solution will take careful analysis and does not sound like a fit candidate for backpatching. I left the catalog infrastructure in place and just reduced the transform functions to always-return-NULL. This would be necessary anyway in the back branches, and it doesn't seem important to be more invasive in HEAD. Bug introduced by commit b8a18ad48. Back-patch to 9.5 where that came in. Report: https://postgr.es/m/20170118144828.1432.52823@wrigleys.postgresql.org Discussion: https://postgr.es/m/18771.1484759439@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/c22ecc6562aac895f0f0529707d7bdb460fd2a49
  • Reset the proper GUC in create_index test. Thinko in commit a4523c5aa. It doesn't really affect anything at present, but it would be a problem if any tests added later in this file ought to get index-only-scan plans. Back-patch, like the previous commit, just to avoid surprises in case we add such a test and then back-patch it. Nikita Glukhov Discussion: https://postgr.es/m/8b70135d-ad38-bdd8-ac92-71e2b3c273cf@postgrespro.ru http://git.postgresql.org/pg/commitdiff/1586317c3f57e619e0cde674c6da406f9d73aaff
  • Doc: improve documentation of new SRF-in-tlist behavior. Correct a misstatement about how things used to work: we did allow nested SRFs before, as long as no function had more than one set-returning input. Also, attempt to document the fact that the new implementation changes the behavior for SRFs within conditional constructs (eg CASE): the conditional construct no longer gates whether the SRF is run, and thus cannot affect the number of rows emitted. We might want to change this behavior, but first it behooves us to see if we can explain it. Minor other wordsmithing on what I wrote yesterday, too. Discussion: https://postgr.es/m/20170118214702.54b2mdbxce5piwv5@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/f13a1277aa2df301583c6db9c2989d2e9d7f6483
  • Fix Assert failure induced by commit 215b43cdc. I'd somehow talked myself into believing that set_append_rel_size doesn't need to worry about getting back an AND clause when it applies eval_const_expressions to the result of adjust_appendrel_attrs (that is, transposing the appendrel parent's restriction clauses for one child). But that is nonsense, and Andreas Seltenreich's fuzz tester soon turned up a counterexample. Put back the make_ands_implicit step that was there before, and add a regression test covering the case. Report: https://postgr.es/m/878tq6vja6.fsf@ansel.ydns.eu http://git.postgresql.org/pg/commitdiff/d479e37e3d20efad8b178e0f1e468c086a7519a8
  • Avoid core dump for empty prepared statement in an aborted transaction. Brown-paper-bag bug in commit ab1f0c822: the old code here coped with null CachedPlanSource.raw_parse_tree, the new code not so much. Per report from Dave Cramer. No regression test, because our core testing infrastructure doesn't provide any easy way to exercise this path. Fortunately, the JDBC crew test it regularly. Discussion: https://postgr.es/m/CADK3HH+Ug3xCysKqw_dZOnaNnytZ1Rh5yP05hjO-e4NoyRxVvA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/ba61a04bc7fefeee03416d9911eb825c4897c223
  • Allow backslash line continuations in pgbench's meta commands. A pgbench meta command can now be continued onto additional line(s) of a script file by writing backslash-return. The continuation marker is equivalent to white space in that it separates tokens. Eventually it'd be nice to have the same thing in psql, but that will be a much larger project. Fabien Coelho, reviewed by Rafia Sabih Discussion: https://postgr.es/m/alpine.DEB.2.20.1610031049310.19411@lancre http://git.postgresql.org/pg/commitdiff/cdc2a70470bdbe3663dc464deb753d6d931bba61
  • Try to fix non-MSVC Windows builds in the wake of logical replication. pgoutput evidently needs to be built without -DBUILDING_DLL. (It seems like a pretty bad idea that these makefiles need to know exactly where all the shlibs are in the tree, or maybe what's bad is putting them under src/backend/. But right now is not the time to redesign that.) Also, remove "override CPPFLAGS" in pgoutput's Makefile. I don't think that that actually has any bad consequences, but it's certainly useless in a directory that has no .h files, and it might be contributing to the failure somehow. Per buildfarm. http://git.postgresql.org/pg/commitdiff/0502e854640bd024f349c0df46b7dd6812b8c05c
  • Fix cross-shlib linking in temporary installs on HPUX 10. Turns out this has been broken for years and we'd not noticed. The one case that was getting exercised in the buildfarm, or probably anywhere else, was postgres_fdw.sl's reference to libpq.sl; and it turns out that that was always going to libpq.sl in the actual installation directory not the temporary install. We'd not noticed because the buildfarm script does "make install" before it tests contrib. However, the recent addition of a logical-replication test to the core regression scripts resulted in trying to use libpqwalreceiver.sl before "make install" happens, and that failed for lack of finding libpq.sl, as shown by failures on buildfarm members gaur and pademelon. There are two changes needed to fix it: the magic environment variable to specify shlib search path at runtime is SHLIB_PATH not LD_LIBRARY_PATH, and the shlib link command needs to specify the +s switch else the library will not honor SHLIB_PATH. I'm not quite sure why buildfarm members anole and gharial (HPUX 11) didn't show the same failure. Consulting man pages on the web says that HPUX 11 honors both LD_LIBRARY_PATH and SHLIB_PATH, which would explain half of it, and the rather confusing wording I've been able to find suggests that +s might effectively be the default in HPUX 11. But it seems at least as likely that there's just a libpq.so installed in /usr/lib on that machine; as long as it's not too ancient, that would satisfy the test. In any case I do not think this patch will break HPUX 11. At the moment I don't see a need to back-patch this, since it only matters for testing purposes, not to mention that HPUX 10 is probably dead in the real world anyway. http://git.postgresql.org/pg/commitdiff/d2ab1176160e30543da1e48f7e0d17564852b693
  • Remove no-longer-needed loop in ExecGather(). Coverity complained quite properly that commit ea15e1867 had introduced unreachable code into ExecGather(); to wit, it was no longer possible to iterate the final for-loop more or less than once. So remove the for(). In passing, clean up a couple of comments, and make better use of a local variable. http://git.postgresql.org/pg/commitdiff/0a8b9d3b2c57028f7100078cd711370f396d5a81
  • Relocate static function declarations to be after typedefs in jsonfuncs.c. Project style is to put things in this order, for the good and sufficient reason that you often need the typedefs in the function declarations. There already was one function declaration that needed a typedef, which was randomly placed away from all the other static function declarations in consequence. And the submitted patch for better json_populate_record functionality jumped through even more hoops in order to preserve this bad idea. This patch only moves lines from point A to point B, no other changes. http://git.postgresql.org/pg/commitdiff/90992e0e2f9fc4aa0f6402f0327604e5fef4630c

Peter Eisentraut pushed:

Álvaro Herrera pushed:

Robert Haas pushed:

Andres Freund pushed:

  • Move targetlist SRF handling from expression evaluation to new executor node. Evaluation of set returning functions (SRFs_ in the targetlist (like SELECT generate_series(1,5)) so far was done in the expression evaluation (i.e. ExecEvalExpr()) and projection (i.e. ExecProject/ExecTargetList) code. This meant that most executor nodes performing projection, and most expression evaluation functions, had to deal with the possibility that an evaluated expression could return a set of return values. That's bad because it leads to repeated code in a lot of places. It also, and that's my (Andres's) motivation, made it a lot harder to implement a more efficient way of doing expression evaluation. To fix this, introduce a new executor node (ProjectSet) that can evaluate targetlists containing one or more SRFs. To avoid the complexity of the old way of handling nested expressions returning sets (e.g. having to pass up ExprDoneCond, and dealing with arguments to functions returning sets etc.), those SRFs can only be at the top level of the node's targetlist. The planner makes sure (via split_pathtarget_at_srfs()) that SRF evaluation is only necessary in ProjectSet nodes and that SRFs are only present at the top level of the node's targetlist. If there are nested SRFs the planner creates multiple stacked ProjectSet nodes. The ProjectSet nodes always get input from an underlying node. We also discussed and prototyped evaluating targetlist SRFs using ROWS FROM(), but that turned out to be more complicated than we'd hoped. While moving SRF evaluation to ProjectSet would allow to retain the old "least common multiple" behavior when multiple SRFs are present in one targetlist (i.e. continue returning rows until all SRFs are at the end of their input at the same time), we decided to instead only return rows till all SRFs are exhausted, returning NULL for already exhausted ones. We deemed the previous behavior to be too confusing, unexpected and actually not particularly useful. As a side effect, the previously prohibited case of multiple set returning arguments to a function, is now allowed. Not because it's particularly desirable, but because it ends up working and there seems to be no argument for adding code to prohibit it. Currently the behavior for COALESCE and CASE containing SRFs has changed, returning multiple rows from the expression, even when the SRF containing "arm" of the expression is not evaluated. That's because the SRFs are evaluated in a separate ProjectSet node. As that's quite confusing, we're likely to instead prohibit SRFs in those places. But that's still being discussed, and the code would reside in places not touched here, so that's a task for later. There's a lot of, now superfluous, code dealing with set return expressions around. But as the changes to get rid of those are verbose largely boring, it seems better for readability to keep the cleanup as a separate commit. Author: Tom Lane and Andres Freund Discussion: https://postgr.es/m/20160822214023.aaxz5l4igypowyri@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/69f4b9c85f168ae006929eec44fc44d569e846b9
  • Adapt python regression tests to 69f4b9c85f16. Hopefully this'll unbreak the buildfarm. http://git.postgresql.org/pg/commitdiff/8b07aee8c5d803801c00103f0d61e32356aaf29c
  • Remove obsoleted code relating to targetlist SRF evaluation. Since 69f4b9c plain expression evaluation (and thus normal projection) can't return sets of tuples anymore. Thus remove code dealing with that possibility. This will require adjustments in external code using ExecEvalExpr()/ExecProject() - that should neither be hard nor very common. Author: Andres Freund and Tom Lane Discussion: https://postgr.es/m/20160822214023.aaxz5l4igypowyri@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/ea15e18677fc2eff3135023e27f69ed8821554ed
  • Fix platform dependant regression output triggered by 69f4b9c85f16. Due to the changed costing in that commit hash-aggregates started to be used, which results in big-endian vs. little-endian output differences. Disable hash-aggs for those tests. Author: Andres Freund, with input from Tom Lane Discussion: https://postgr.es/m/22891.1484791792@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/182200531a92204b0547d337f50b665d222af168

Stephen Frost pushed:

  • Dump sequence data based on the TableDataInfo flag. When considering a sequence's Data entry in dumpSequenceData, we were actually looking at the sequence definition's dump flag to decide if we should dump the data or not. That's generally fine, except for when the sequence data entry was created by processExtensionTables() because it's a config sequence. In that case, the sequence itself won't be marked as dumping data because it's part of an extension, leading to the need for processExtensionTables() to create the sequence data entry. This leads to extension config sequence data not being included in the dump when it should be. Fix this by looking at the sequence data's dump flag instead, just as dumpTableData() was doing for tables (which is why config tables were correctly being handled), and add a regression test to make sure we don't break it moving forward. All of this is a bit round-about since we can now represent which components of a given dump item should be dumped out through the dump flag. A future improvement might be to change checkExtensionMembership() to check for config sequences/tables and set the dump flag based on that directly, possibly removing the need for processExtensionTables(). Bug found by Daniele Varrazzo. Discussion: https://postgr.es/m/CA+mi_8ZmxQM7+nZ7pJ8uyfxc9V3o=UAG14dVqvftdmvw8OJ3gQ@mail.gmail.com Patch by Michael Paquier, with some tweaking of the regression tests by me. Back-patch to 9.6 where the bug was introduced. http://git.postgresql.org/pg/commitdiff/bec96c82f8ff4fcf7ef0f070f6f7447edf106de3

Correctifs en attente

Michaël Paquier and Karl O. Pinc traded patches to implement pg_current_logfile().

Michaël Paquier sent in another revision of a patch to rename pg_clog to pg_xact and pg_subtrans to pg_subxact.

Masahiko Sawada sent in two more revisions of a patch to support FDW transactions.

Amit Kapila sent in another revision of a patch to parallelize queries containing subplans.

Haribabu Kommi sent in four more revisions of a patch to implement a pg_hba_file_settings view.

Beena Emerson and Robert Haas traded patches to enable increasing WAL segment size.

Amit Kapila and Robert Haas traded patches to enable parallel index scans.

Michaël Paquier sent in a flock of back-patches for an issue in backups on standbys uncovered by sqlsmith.

Mithun Cy sent in another revision of a patch to cache hash index meta pages.

Nikita Glukhov sent in a patch to implement K-nearest-neighbor for B-Tree indexes.

Dilip Kumar sent in another revision of a patch to implement parallel bitmap heap scan.

Magnus Hagander and Michaël Paquier traded patches to implement a version of jsonb_delete which takes arrays as input.

Haribabu Kommi sent in another revision of a patch to implement a pg_stat_sql view.

Etsuro Fujita sent in another revision of a patch to fix a bug in foreign joins.

David Rowley sent in two more revisions of a patch to improve performance for outer joins where the outer side is unique.

Haribabu Kommi sent in another revision of a patch to implement a 64-bit macaddr type.

Ashutosh Sharma sent in another revision of a patch to add pgstathashindex to the pgstattuple extension.

Ashutosh Sharma and Jesper Pedersen traded patches to add hash index support to pageinspect.

Petr Jelínek sent in two more revisions of a patch to enable logical replication support for the initial data copy.

Petr Jelínek sent in another revision of a patch to add RENAME support for PUBLICATIONs and SUBSCRIPTIONs.

Petr Jelínek sent in a flock of patches to fix issues in logical replication.

Rafia Sabih sent in another revision of a patch to implement parallel index-only scans.

Pavan Deolasee sent in another revision of a patch to implement WARM.

Dmitry Fedin sent in a patch to change a misleading comment in pgtime.h.

Yugo Nagata sent in a patch to fix a comment in freelist.c.

Erik Rijkers sent in a patch to improve the logical replication documentation.

Jim Nasby sent in a patch to protect syscache from bloating with negative cache entries.

Petr Jelínek sent in a patch to fix AFTER trigger execution in logical replication.

Tom Lane sent in another revision of a patch to prevent the possibility of UNKNOWN output columns.

par N Bougain le jeudi 2 février 2017 à 00h43

jeudi 19 janvier 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 15 janvier 2017

[ndt: Meetup à Toulouse le 2 mars :https://www.meetup.com/fr-FR/PostgreSQL-User-Group-Toulouse/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170116044539.GA14726@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Magnus Hagander pushed:

Tom Lane pushed:

  • Expand the regression tests for PL/Tcl. This raises the test coverage (by line count) in pltcl.c from about 70% to 86%. Karl Lehenbauer and Jim Nasby Discussion: https://postgr.es/m/92a1670d-21b6-8f03-9c13-e4fb2207ab7b@BlueTreble.com http://git.postgresql.org/pg/commitdiff/961bed0208912a929a47c5a30190ff76748f3a03
  • Fix error handling in pltcl_returnnext. We can't throw elog(ERROR) out of a Tcl command procedure; we have to catch the error and return TCL_ERROR to the Tcl interpreter. pltcl_returnnext failed to meet this requirement, so that errors detected by pltcl_build_tuple_result or other functions called here led to longjmp'ing out of the Tcl interpreter and thereby leaving it in a bad state. Use the existing subtransaction support to prevent that. Oversight in commit 26abb50c4, found more or less accidentally by the buildfarm thanks to the tests added in 961bed020. Report: https://postgr.es/m/30647.1483989734@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/8c5722948e831c1862a39da2bb5d793a6f2aabab
  • Fix field order in struct catcache. Somebody failed to grasp the point of having the #ifdef CATCACHE_STATS fields at the end of the struct. Put that back the way it should be, and add a comment making it more explicit why it should be that way. http://git.postgresql.org/pg/commitdiff/e2117e4ab0d7fcb76f20cbf2e52541998d639d9e
  • In PL/Tcl tests, don't choke if optional error fields are missing. This fixes a portability issue introduced by commit 961bed020: with a compiler that doesn't support PG_FUNCNAME_MACRO, the "funcname" field of errorCode won't be provided, leading to a failure of the unset command. I added -nocomplain to the unset commands for filename and lineno too, just in case, though I know of no platform that wouldn't populate those fields. (BTW, -nocomplain is new in Tcl 8.4, but fortunately we dropped support for pre-8.4 Tcl some time ago.) Per buildfarm member pademelon. http://git.postgresql.org/pg/commitdiff/5b29e6b688d1e783a098aa24f9e795f8de798a87
  • Fix some more regression test row-order-instability issues. Commit 0563a3a8b just introduced another instance of the same unsafe testing methodology that appeared in 2ac3ef7a0, which I corrected in 257d81572. Robert/Amit, please stop doing that. Also look through the rest of f0e44751d's test cases, and correct some other queries with underdetermined ordering of results from the system catalogs. These haven't failed in the buildfarm yet, but I don't have any confidence in that staying true. Per multiple buildfarm members. http://git.postgresql.org/pg/commitdiff/5ad966ab1c50e829462f2b3e3ffa59e2d95479e6
  • Throw suitable error for COPY TO STDOUT/FROM STDIN in a SQL function. A client copy can't work inside a function because the FE/BE wire protocol doesn't support nesting of a COPY operation within query results. (Maybe it could, but the protocol spec doesn't suggest that clients should support this, and libpq for one certainly doesn't.) In most PLs, this prohibition is enforced by spi.c, but SQL functions don't use SPI. A comparison of _SPI_execute_plan() and init_execution_state() shows that rejecting client COPY is the only discrepancy in what they allow, so there's no other similar bugs. This is an astonishingly ancient oversight, so back-patch to all supported branches. Report: https://postgr.es/m/BY2PR05MB2309EABA3DEFA0143F50F0D593780@BY2PR05MB2309.namprd05.prod.outlook.com http://git.postgresql.org/pg/commitdiff/75abb955dfef064f2fbc5c043f37fff8d0262ffe
  • Change representation of statement lists, and add statement location info. This patch makes several changes that improve the consistency of representation of lists of statements. It's always been the case that the output of parse analysis is a list of Query nodes, whatever the types of the individual statements in the list. This patch brings similar consistency to the outputs of raw parsing and planning steps: * The output of raw parsing is now always a list of RawStmt nodes; the statement-type-dependent nodes are one level down from that. * The output of pg_plan_queries() is now always a list of PlannedStmt nodes, even for utility statements. In the case of a utility statement, "planning" just consists of wrapping a CMD_UTILITY PlannedStmt around the utility node. This list representation is now used in Portal and CachedPlan plan lists, replacing the former convention of intermixing PlannedStmts with bare utility-statement nodes. Now, every list of statements has a consistent head-node type depending on how far along it is in processing. This allows changing many places that formerly used generic "Node *" pointers to use a more specific pointer type, thus reducing the number of IsA() tests and casts needed, as well as improving code clarity. Also, the post-parse-analysis representation of DECLARE CURSOR is changed so that it looks more like EXPLAIN, PREPARE, etc. That is, the contained SELECT remains a child of the DeclareCursorStmt rather than getting flipped around to be the other way. It's now true for both Query and PlannedStmt that utilityStmt is non-null if and only if commandType is CMD_UTILITY. That allows simplifying a lot of places that were testing both fields. (I think some of those were just defensive programming, but in many places, it was actually necessary to avoid confusing DECLARE CURSOR with SELECT.) Because PlannedStmt carries a canSetTag field, we're also able to get rid of some ad-hoc rules about how to reconstruct canSetTag for a bare utility statement; specifically, the assumption that a utility is canSetTag if and only if it's the only one in its list. While I see no near-term need for relaxing that restriction, it's nice to get rid of the ad-hocery. The API of ProcessUtility() is changed so that what it's passed is the wrapper PlannedStmt not just the bare utility statement. This will affect all users of ProcessUtility_hook, but the changes are pretty trivial; see the affected contrib modules for examples of the minimum change needed. (Most compilers should give pointer-type-mismatch warnings for uncorrected code.) There's also a change in the API of ExplainOneQuery_hook, to pass through cursorOptions instead of expecting hook functions to know what to pick. This is needed because of the DECLARE CURSOR changes, but really should have been done in 9.6; it's unlikely that any extant hook functions know about using CURSOR_OPT_PARALLEL_OK. Finally, teach gram.y to save statement boundary locations in RawStmt nodes, and pass those through to Query and PlannedStmt nodes. This allows more intelligent handling of cases where a source query string contains multiple statements. This patch doesn't actually do anything with the information, but a follow-on patch will. (Passing this information through cleanly is the true motivation for these changes; while I think this is all good cleanup, it's unlikely we'd have bothered without this end goal.) catversion bump because addition of location fields to struct Query affects stored rules. This patch is by me, but it owes a good deal to Fabien Coelho who did a lot of preliminary work on the problem, and also reviewed the patch. Discussion: https://postgr.es/m/alpine.DEB.2.20.1612200926310.29821@lancre http://git.postgresql.org/pg/commitdiff/ab1f0c8225714aaa18d2f9ca4f80cd009f145421
  • Teach contrib/pg_stat_statements to handle multi-statement commands better. Make use of the statement boundary info added by commit ab1f0c822 to let pg_stat_statements behave more sanely when multiple SQL queries are jammed into one query string. It now records just the relevant part of the source string, not the whole thing, for each individual query. Even when no multi-statement strings are involved, users may notice small changes in the output: leading and trailing whitespace and semicolons will be stripped from statements, which did not happen before. Also, significantly expand pg_stat_statements' regression test script. Fabien Coelho, reviewed by Craig Ringer and Kyotaro Horiguchi, some mods by me Discussion: https://postgr.es/m/alpine.DEB.2.20.1612200926310.29821@lancre http://git.postgresql.org/pg/commitdiff/83f2061dd037477ec8479ee160367840e203a722
  • Fix matching of boolean index columns to sort ordering. Normally, if we have a WHERE clause like "indexcol = constant", the planner will figure out that that index column can be ignored when determining whether the index has a desired sort ordering. But this failed to work for boolean index columns, because a condition like "boolcol = true" is canonicalized to just "boolcol" which does not give rise to an EquivalenceClass. Add a check to allow the same type of deduction to be made in this case too. Per a complaint from Dima Pavlov. Arguably this is a bug, but given the limited impact and the small number of complaints so far, I won't risk destabilizing plans in stable branches by back-patching. Patch by me, reviewed by Michael Paquier Discussion: https://postgr.es/m/1788.1481605684@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/0777f7a2e8e0a51f0f60cfe164d538bb459bf9f2

Ãlvaro Herrera pushed:

  • BRIN revmap pages are not standard pages, and therefore we ought not to tell XLogRegisterBuffer the opposite, when writing XLog for a brin update that moves the index tuple to a different page. Otherwise, xlog insertion would try to "compress the hole" when producing a full-page image for it; but since we don't update pd_lower/upper, the hole covers the whole page. On WAL replay, the revmap page becomes empty and so the entire portion of the index is useless and needs to be recomputed. This is low-probability: a BRIN update only moves an index tuple to a different page when the summary tuple is larger than the existing one, which doesn't happen with fixed-width datatypes. Also, the revmap page must be first after a checkpoint. Report and patch: Kuntal Ghosh Bug is alleged to have detected by a WAL-consistency-checking tool. Discussion: https://postgr.es/m/CAGz5QCJ=00UQjScSEFbV=0qO5ShTZB9WWz_Fm7+Wd83zPs9Geg@mail.gmail.com I posted a test case demonstrating the problem, but I'm refraining from adding it to the test suite; if the WAL consistency tool makes it in, that will be a better way to catch this from regressing. (We should definitely have someting that causes not-same-page updates, though.) http://git.postgresql.org/pg/commitdiff/7403561c0f6a8c62b79b6ddf0364ae6c01719068
  • Fix ALTER TABLE / SET TYPE for irregular inheritance. If inherited tables don't have exactly the same schema, the USING clause in an ALTER TABLE / SET DATA TYPE misbehaves when applied to the children tables since commit 9550e8348b79. Starting with that commit, the attribute numbers in the USING expression are fixed during parse analysis. This can lead to bogus errors being reported during execution, such as: ERROR: attribute 2 has wrong type DETAIL: Table has type smallint, but query expects integer. Since it wouldn't do to revert to the original coding, we now apply a transformation to map the attribute numbers to the correct ones for each child. Reported by Justin Pryzby Analysis by Tom Lane; patch by me. Discussion: https://postgr.es/m/20170102225618.GA10071@telsasoft.com http://git.postgresql.org/pg/commitdiff/3957b58b8885441c8d03bc1cfc00e47cf8cd7975
  • Fix overflow check in StringInfo; add missing casts. A few thinkos I introduced in fa2fa9955280. Also, amend a similarly broken comment. Report by Daniel Vérité. Authors: Daniel Vérité, Ãlvaro Herrera Discussion: https://postgr.es/m/1706e85e-60d2-494e-8a64-9af1e1b2186e@manitou-mail.org http://git.postgresql.org/pg/commitdiff/42f50cb8fa9848bbbc6776bcea03293a6b28b2d4

Stephen Frost pushed:

  • Fix invalid-parallel-jobs error message. Including the program name twice is not helpful: -> pg_dump -j -1 pg_dump: pg_dump: invalid number of parallel jobs Correct by removing the progname from the exit_horribly() call used when validating the number of parallel jobs. Noticed while testing various pg_dump error cases. Back-patch to 9.3 where parallel pg_dump was added. http://git.postgresql.org/pg/commitdiff/2ef6fe9cbae9fe7789a35cbc5fa1bbf78c163d42
  • pg_dump: Strict names with no matching schema. When using pg_dump --strict-names and a schema pattern which doesn't match any schemas (eg: --schema='nonexistant*'), we were incorrectly throwing an error claiming no tables were found when, really, there were no schemas found: -> pg_dump --strict-names --schema='nonexistant*' pg_dump: no matching tables were found for pattern "nonexistant*" Fix that by changing the error message to say 'schemas' instead, since that is what we are actually complaining about. Noticed while testing pg_dump error cases. Back-patch to 9.6 where --strict-names and this error message were introduced. http://git.postgresql.org/pg/commitdiff/abfd0095c1e1a2e3fad2696516b64871895334ec
  • pg_restore: Don't allow non-positive number of jobs. pg_restore will currently accept invalid values for the number of parallel jobs to run (eg: -1), unlike pg_dump which does check that the value provided is reasonable. Worse, '-1' is actually a valid, independent, parameter (as an alias for --single-transaction), leading to potentially completely unexpected results from a command line such as: -> pg_restore -j -1 Where a user would get neither parallel jobs nor a single-transaction. Add in validity checking of the parallel jobs option, as we already have in pg_dump, before we try to open up the archive. Also move the check that we haven't been asked to run more parallel jobs than possible on Windows to the same place, so we do all the option validity checking before opening the archive. Back-patch all the way, though for 9.2 we're adding the Windows-specific check against MAXIMUM_WAIT_OBJECTS as that check wasn't back-patched originally. Discussion: https://www.postgresql.org/message-id/20170110044815.GC18360%40tamriel.snowman.net http://git.postgresql.org/pg/commitdiff/e72059f3757594c5530ce321acdbe67f0da5da13

Robert Haas pushed:

  • Improve coding in _hash_addovflpage. Instead of relying on the page contents to know whether we have advanced from the primary bucket page to an overflow page, track that explicitly. Amit Kapila, per a complaint by me. http://git.postgresql.org/pg/commitdiff/e898437460f55b49623d1aea435cd92e0011d54d
  • Fix incorrect function name in comment. Amit Langote http://git.postgresql.org/pg/commitdiff/76568d37865c5c21ae154008b2c681e3e32ac880
  • Fix cardinality estimates for parallel joins. For a partial path, the cardinality estimate needs to reflect the number of rows we think each worker will see, rather than the total number of rows; otherwise, costing will go wrong. The previous coding got this completely wrong for parallel joins. Unfortunately, this change may destabilize plans for users of 9.6 who have enabled parallel query, but since 9.6 is still fairly new I'm hoping expectations won't be too settled yet. Also, this is really a brown-paper-bag bug, so leaving it unfixed for the entire lifetime of 9.6 seems unwise. Related reports (whose import I initially failed to recognize) by Tomas Vondra and Tom Lane. Discussion: http://postgr.es/m/CA+TgmoaDxZ5z5Kw_oCQoymNxNoVaTCXzPaODcOuao=CzK8dMZw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/0c2070cefa0e5d097b715c9a3b9b5499470019aa
  • Fix a bug in how we generate partition constraints. Move the code for doing parent attnos to child attnos mapping for Vars in partition constraint expressions to a separate function map_partition_varattnos() and call it from the appropriate places. Doing it in get_qual_from_partbound(), as is now, would produce wrong result in certain multi-level partitioning cases, because it only considers the current pair of parent-child relations. In certain multi-level partitioning cases, attnums for the same key attribute(s) might differ between various levels causing the same attribute to be numbered differently in different instances of the Var corresponding to a given attribute. With this commit, in generate_partition_qual(), we first generate the the whole partition constraint (considering all levels of partitioning) and then do the mapping, so that Vars in the final expression are numbered according the leaf relation (to which it is supposed to apply). Amit Langote, reviewed by me. http://git.postgresql.org/pg/commitdiff/0563a3a8b59150bf3cc8b2b7077f684e0eaf8aff

Bruce Momjian pushed:

Peter Eisentraut pushed:

Correctifs en attente

Amul Sul sent in two more revisions of a patch to implement pg_background.

Dilip Kumar sent in four more revisions of a patch to implement parallel bitmap heap scan.

Euler Taveira de Oliveira sent in another revision of a patch to move collation import to the backend.

Anastasia Lubennikova and Erik Rijkers traded patches to implement covering + unique indexes.

Takayuki Tsunakawa sent in another revision of a patch to support huge pages on Windows.

Jon Nelson sent in another revision of a patch to guc-ify the formerly hard-coded MAX_SEND_SIZE to max_wal_send.

Ãlvaro Herrera sent in another revision of a patch to help fix pg_dump and COPY bugs around big lines.

David Fetter sent in three more revisions of a patch to add an extension that disallows simple UPDATEs and DELETEs that lack any WHERE clause.

Ãlvaro Herrera sent in a patch to remove floats from the bootstrap scanner and parser.

Antonin Houska sent in two revisions of a patch to implement grouped base relation, which is infrastructure for distributed aggregation.

Vladimir Rusinov sent in four more revisions of a patch to rename things *xlog* to things *wal*.

Keith Fiske and Amit Langote traded patches to fix up declarative partitioning.

Haribabu Kommi sent in another revision of a patch to add macaddr 64 bit (EUI-64) datatype support.

Michaël Paquier sent in two more revisions of a patch to add compression-level adjustment to pg_receivelog.

Kyotaro HORIGUCHI sent in another revision of a patch to implement radix trees for character conversion.

Ashutosh Sharma sent in another revision of a patch to add microvacuum support for hash indexes.

Pavel Stěhule sent in another revision of a patch to implement \gstore and friends in psql.

Matheus de Oliveira sent in another revision of a patch to add ALTER DEFAULT PRIVILEGES with GRANT/REVOKE ON SCHEMAS.

Rafia Sabih sent in two revisions of a patch to enable passing query string to workers.

Jesper Pedersen and Ashutosh Sharma traded patches to add hash index support to the pageinspect contrib extension.

Nikita Glukhovs sent in another revision of a patch to add a recursive version of json_populate_record().

David Rowley sent in a patch to make connection_limit ignore bgworkers.

Peter Eisentraut sent in a patch to factor out the many copies of atooid.

Robert Haas sent in another revision of a patch to implement gather merge.

Etsuro Fujita sent in a patch to rearrange some function declarations for correctness.

Pavel Stěhule sent in a patch to add some checks to PL/pgsql.

Fabien COELHO and Rafia Sabih traded patches to give pgbench a way to use backslash as a continuation character.

Ãlvaro Herrera and Pavel StÄ›hule traded patches to implement xmltable().

Amit Kapila sent in another revision of a patch to add WAL logging for hash indexes.

Mithun Cy sent in another revision of a patch to cache hash index meta pages.

Etsuro Fujita sent in another revision of a patch to push more FULL JOINs down to FDWs.

Ashutosh Bapat sent in a patch to remove an unused member root in foreign_glob_cxt.

Dmitry Dolgov sent in another revision of a patch to implement generic type subscripts.

Gilles Darold and Karl O. Pinc traded patches to implement pg_current_logfile().

Rafia Sabih sent in another revision of a patch to add parallel index-only scan.

Mithun Cy sent in a patch to fix a typo in a comment in hashsearch.c.

Ants Aasma sent in another revision of a patch to send hot standby feedback on first connect.

Anastasia Lubennikova sent in a patch to implement IF NOT EXISTS option for CREATE SERVER and CREATE USER MAPPING statements.

Amit Khandekar sent in a patch to prevent useless VACUUMs.

Peter Moser sent in another revision of a patch to implement temporal query processing with range types.

Etsuro Fujita sent in another revision of a patch to fix a bug in the Postgres FDW.

Amit Kapila sent in another revision of a patch to parallelize queries containing subplans.

Petr Jelínek sent in two more revisions of a patch to implement logical replication.

par N Bougain le jeudi 19 janvier 2017 à 22h47

dimanche 15 janvier 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 8 janvier 2017

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en janvier

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170109204928.GA13529@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • In pgbench logging, avoid assuming that instr_times match Unix timestamps. For aggregated logging, pg_bench supposed that printing the integer part of INSTR_TIME_GET_DOUBLE() would produce a Unix timestamp. That was already broken on Windows, and it's about to get broken on most other platforms as well. As in commit 74baa1e3b, we can remove the entanglement at the price of one extra syscall per transaction; though here it seems more convenient to use time(NULL) instead of gettimeofday(), since we only need integral-second precision. I took the time to do some wordsmithing on the documentation about pgbench's logging features, too. Discussion: https://postgr.es/m/8837.1483216839@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/67a875355e4a6db294e9652af5a883876ddeb4a5
  • Use clock_gettime(), if available, in instr_time measurements. The advantage of clock_gettime() is that the API allows the result to be precise to nanoseconds, not just microseconds as in gettimeofday(). Now that it's routinely possible to do tens of plan node executions in 1us, we really need more precision than gettimeofday() can offer for EXPLAIN ANALYZE to accumulate statistics with. Some research shows that clock_gettime() is available on pretty nearly every modern Unix-ish platform, and as far as I have been able to test, it has about the same execution time as gettimeofday(), so there's no loss in switching over. (By the same token, this doesn't do anything to fix the fact that we really wish clock readings were faster. But there's enough win here to justify changing anyway.) A small side benefit is that on most platforms, we can use CLOCK_MONOTONIC instead of CLOCK_REALTIME and thereby render EXPLAIN impervious to concurrent resets of the system clock. (This means that code must not assume that the contents of struct instr_time have any well-defined interpretation as timestamps, but really that was true before.) Some platforms offer nonstandard clock IDs that might be of interest. This patch knows we should use CLOCK_MONOTONIC_RAW on macOS, because it provides more precision and is faster to read than their CLOCK_MONOTONIC. If there turn out to be many more cases where we need special rules, it might be appropriate to handle the selection of clock ID in configure, but for the moment that doesn't seem worth the trouble. Discussion: https://postgr.es/m/31856.1400021891@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/1d63f7d2d180c8708bc12710254eb7b45823440f
  • Allow SSL configuration to be updated at SIGHUP. It is no longer necessary to restart the server to enable, disable, or reconfigure SSL. Instead, we just create a new SSL_CTX struct (by re-reading all relevant files) whenever we get SIGHUP. Testing shows that this is fast enough that it shouldn't be a problem. In conjunction with that, downgrade the logic that complains about pg_hba.conf "hostssl" lines when SSL isn't active: now that's just a warning condition not an error. An issue that still needs to be addressed is what shall we do with passphrase-protected server keys? As this stands, the server would demand the passphrase again on every SIGHUP, which is certainly impractical. But the case was only barely supported before, so that does not seem a sufficient reason to hold up committing this patch. Andreas Karlsson, reviewed by Michael Banck and Michael Paquier Discussion: https://postgr.es/m/556A6E8A.9030400@proxel.se http://git.postgresql.org/pg/commitdiff/de41869b64d57160f58852eab20a27f248188135
  • Disable prompting for passphrase while (re)loading SSL config files. OpenSSL's default behavior when loading a passphrase-protected key file is to open /dev/tty and demand the password from there. It was kinda sorta okay to allow that to happen at server start, but really that was never workable in standard daemon environments. And it was a complete fail on Windows, where the same thing would happen at every backend launch. Yesterday's commit de41869b6 put the final nail in the coffin by causing that to happen at every SIGHUP; even if you've still got a terminal acting as the server's TTY, having the postmaster freeze until you enter the passphrase again isn't acceptable. Hence, override the default behavior with a callback that returns an empty string, ensuring failure. Change the documentation to say that you can't have a passphrase-protected server key, period. If we can think of a production-grade way of collecting a passphrase from somewhere, we might do that once at server startup and use this callback to feed it to OpenSSL, but it's far from clear that anyone cares enough to invest that much work in the feature. The lack of complaints about the existing fractionally-baked behavior suggests nobody's using it anyway. Discussion: https://postgr.es/m/29982.1483412575@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/1e942c7474d7b5e4bfa04918d2f68d95902f17b3
  • Re-allow SSL passphrase prompt at server start, but not thereafter. Leave OpenSSL's default passphrase collection callback in place during the first call of secure_initialize() in server startup. Although that doesn't work terribly well in daemon contexts, some people feel we should not break it for anyone who was successfully using it before. We still block passphrase demands during SIGHUP, meaning that you can't adjust SSL configuration on-the-fly if you used a passphrase, but this is no worse than what it was before commit de41869b6. And we block passphrase demands during EXEC_BACKEND reloads; that behavior wasn't useful either, but at least now it's documented. Tweak some related log messages for more readability, and avoid issuing essentially duplicate messages about reload failure caused by a passphrase. Discussion: https://postgr.es/m/29982.1483412575@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/6667d9a6d77b9a6eac89638ac363b6d03da253c1
  • Prefer int-wide pg_atomic_flag over char-wide when using gcc intrinsics. configure can only probe the existence of gcc intrinsics, not how well they're implemented, and unfortunately the answer is sometimes "badly". In particular we've found that multiple compilers fail to implement char-width __sync_lock_test_and_set() correctly on PPC; and even a correct implementation would necessarily be pretty inefficient, since that hardware has only a word-wide primitive to work with. Given the knowledge we've accumulated in s_lock.h, it appears that it's best to rely on int-width TAS operations on most non-Intel architectures. Hence, pick int not char when both are nominally available to us in generic-gcc.h (note that that code is not used for x86[_64]). Back-patch to fix regression test failures on FreeBSD/PPC. Ordinarily back-patching a change like this would be verboten because of ABI breakage. But since pg_atomic_flag is not yet used in any Postgres data structure, there's no ABI to break. It seems safer to back-patch to avoid possible gotchas, if someday we do back-patch something that uses pg_atomic_flag. Discussion: https://postgr.es/m/25414.1483076673@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/698127a4a9bc3c74659bf0e5383b1ed99aeb1570
  • Handle OID column inheritance correctly in ALTER TABLE ... INHERIT. Inheritance operations must treat the OID column, if any, much like regular user columns. But MergeAttributesIntoExisting() neglected to do that, leading to weird results after a table with OIDs is associated to a parent with OIDs via ALTER TABLE ... INHERIT. Report and patch by Amit Langote, reviewed by Ashutosh Bapat, some adjustments by me. It's been broken all along, so back-patch to all supported branches. Discussion: https://postgr.es/m/cb13cfe7-a48c-5720-c383-bb843ab28298@lab.ntt.co.jp http://git.postgresql.org/pg/commitdiff/d86f40009b6b019f794819a9af9038cff0cac6f3
  • Fix handling of empty arrays in array_fill(). array_fill(..., array[0]) produced an empty array, which is probably what users expect, but it was a one-dimensional zero-length array which is not our standard representation of empty arrays. Also, for no very good reason, it rejected empty input arrays; that case should be allowed and produce an empty output array. In passing, remove the restriction that the input array(s) have lower bound 1. That seems rather pointless, and it would have needed extra complexity to make the check deal with empty input arrays. Per bug #14487 from Andrew Gierth. It's been broken all along, so back-patch to all supported branches. Discussion: https://postgr.es/m/20170105152156.10135.64195@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/82f8107b92c9104ec9d9465f3f6a4c6dab4c124a
  • Invalidate cached plans on FDW option changes. This fixes problems where a plan must change but fails to do so, as seen in a bug report from Rajkumar Raghuwanshi. For ALTER FOREIGN TABLE OPTIONS, do this through the standard method of forcing a relcache flush on the table. For ALTER FOREIGN DATA WRAPPER and ALTER SERVER, just flush the whole plan cache on any change in pg_foreign_data_wrapper or pg_foreign_server. That matches the way we handle some other low-probability cases such as opclass changes, and it's unclear that the case arises often enough to be worth working harder. Besides, that gives a patch that is simple enough to back-patch with confidence. Back-patch to 9.3. In principle we could apply the code change to 9.2 as well, but (a) we lack postgres_fdw to test it with, (b) it's doubtful that anyone is doing anything exciting enough with FDWs that far back to need this desperately, and (c) the patch doesn't apply cleanly. Patch originally by Amit Langote, reviewed by Etsuro Fujita and Ashutosh Bapat, who each contributed substantial changes as well. Discussion: https://postgr.es/m/CAKcux6m5cA6rRPTKkqVdJ-R=KKDfe35Q_ZuUqxDSV_4hwga=og@mail.gmail.com http://git.postgresql.org/pg/commitdiff/c52d37c8b3674f1ca9ead297480ade0ac9c54174
  • Merge two copies of tuple-building code in pltcl.c. Make pltcl_trigger_handler() construct modified tuples using pltcl_build_tuple_result(), rather than its own copy of essentially the same logic. This results in slightly different message wording for the error cases, and in one case a different SQLSTATE, but it seems unlikely that any existing applications are depending on any of those details. While at it, fix a typo in commit 26abb50c4: pltcl_build_tuple_result was applying encoding conversion in the wrong direction. That would be a back-patchable bug fix, except the code hasn't shipped yet. Jim Nasby, reviewed by me Discussion: https://postgr.es/m/d2c6425a-d9e0-f034-f774-4a872c234d89@BlueTreble.com http://git.postgresql.org/pg/commitdiff/de5fed0d0c704e3d1a928bf420663319d898cee4
  • Get rid of ParseState.p_value_substitute; use a columnref hook instead. I noticed that p_value_substitute, which is a single-purpose kluge I added in 2002 (commit b0422b215), could be replaced by having domainAddConstraint install a parser hook that looks for the name "value". The parser hook code only dates back to 2009, so it's not surprising that we had to kluge this in 2002, but we can do it more cleanly now. http://git.postgresql.org/pg/commitdiff/7c3abe3c92fd3a14a70bc2f888f936cd6fe28c0f
  • Improve documentation of struct ParseState. I got annoyed about how some fields of ParseState were documented in the struct's block comment and some weren't; not all of the latter are trivial. Fix that. Also reorder a couple of fields that seem to have been placed rather randomly, or maybe with an idea of avoiding padding space; but there are never so many ParseStates in existence at one time that we ought to value pad space over readability. http://git.postgresql.org/pg/commitdiff/3c40594e6eeabb3a8ad22aee93de3a19c41efdc2

Heikki Linnakangas pushed:

Peter Eisentraut pushed:

Bruce Momjian pushed:

Magnus Hagander pushed:

  • Make wal streaming the default mode for pg_basebackup. Since streaming is now supported for all output formats, make this the default as this is what most people want. To get the old behavior, the parameter -X none can be specified to turn it off. This also removes the parameter -x for fetch, now requiring -X fetch to be specified to use that. Reviewed by Vladimir Rusinov, Michael Paquier and Simon Riggs http://git.postgresql.org/pg/commitdiff/9a4d51077c96c10322582211781bb969b51822ff
  • Attempt to handle pending-delete files on Windows. These files are deleted but not yet gone from the filesystem. Operations on them will return ERROR_DELETE_PENDING. With this we start treating that as ENOENT, meaning files does not exist (which is the state it will soon reach). This should be safe in every case except when we try to recreate a file with exactly the same name. This is an operation that PostgreSQL does very seldom, so hopefully that won't happen much -- and even if it does, this treatment should be no worse than treating it as an unhandled error. We've been un able to reproduce the bug reliably, so pushing this to master to get buildfarm coverage and other testing. Once it's proven to be stable, it should be considered for backpatching. Discussion: https://postgr.es/m/20160712083220.1426.58667%40wrigleys.postgresql.org Patch by me and Michael Paquier http://git.postgresql.org/pg/commitdiff/9951741bbeb3ec37ca50e9ce3df1808c931ff6a6

Simon Riggs pushed:

Robert Haas pushed:

Stephen Frost pushed:

  • Protect against NULL-dereference in pg_dump. findTableByOid() is allowed to return NULL and we should therefore be checking for that case. getOwnedSeqs() and dumpSequence() shouldn't ever actually see this happen, but given odd circumstances it might and commit f9e439b1 probably shouldn't have removed that check. Pointed out by Coverity. Initial patch from Michael Paquier. Back-patch to 9.6, where that commit had removed the check. http://git.postgresql.org/pg/commitdiff/d74ecbc8d85eb7a2aa1d5516c5c38d6ab0cbbd82
  • Add basic pg_dumpall/pg_restore TAP tests. For reasons unknown, pg_dumpall and pg_restore managed to escape the basic set of TAP tests that were added for pg_dump in 6bd356c3, so let's get them added now. A few minor adjustments are also made to the dump/restore tests to improve code coverage for pg_restore/pg_dumpall. http://git.postgresql.org/pg/commitdiff/9b815a8ff227e62442259e0fbabc5cf37e433df9

Correctifs en attente

Thomas Munro sent in a patch to add an isolation test for SERIALIZABLE READ ONLY DEFERRABLE.

Ashutosh Bapat sent in another revision of a patch to support partition-wise join between multi-level partitioned tables.

Thomas Munro sent in another revision of a patch to implement causal reads.

Pavan Deolasee sent in two more revisions of a patch to implement WARM.

Ashutosh Bapat sent in a patch to cast nodes more safely.

Thomas Munro sent in another revision of a patch to help measure replay lag.

KaiGai Kohei sent in another revision of a patch to add PassDownLimitBound for ForeignScan/CustomScan.

Mithun Cy sent in two more revisions of a patch to cache hash index meta page.

Amit Kapila sent in another revision of a patch to parallelize queries containing subplans.

Peter Geoghegan sent in another revision of a patch to implement parallel tuplesort.

Tom Lane sent in a patch to expand the existing API to allow the AM to return either a heap or index tuple.

Tomas Vondra sent in another revision of a patch to implement multivariate statistics.

Heikki Linnakangas sent in a patch to replace isMD5() with a more future-proof way to check if password is encrypted and use "plain:" prefix for plaintext passwords stored in pg_authid.

Dilip Kumar sent in another revision of a patch to implement parallel bitmap heap scan.

Craig Ringer sent in two more revisions of a patch to implement logical decoding on standbys.

Peter Eisentraut sent in two revisions of a patch to fix some infelicities in the logical replication protocol and output plugin.

Peter Eisentraut sent in another revision of a patch to add logical replication workers.

Peter Eisentraut and Petr Jelínek traded patches to add PUBLICATION catalogs and DDL.

Ãlvaro Herrera sent in three revisions of a patch to fix and error in ALTER TABLE ... ALTER TYPE.

Haribabu Kommi sent in another revision of a patch to implement a pg_hba_file_settings view and add TAP tests for same.

David Fetter sent in two more revisions of a patch to disable simple UPDATEs and DELETEs which lack any WHERE clause.

Peter Eisentraut sent in another revision of a patch to fix some infelicities between pg_basebackups and slots.

Dilip Kumar sent in another revision of a patch to implement parallel merge join.

Kuntal Ghosh sent in another revision of a patch to implement a WAL consistency check facility.

Craig Ringer sent in another revision of a patch to add PostgresNode methods to wait for node catchup and expand streaming replication tests to cover hot standby feedback and physical replication slots.

Michaël Paquier sent in another revision of a patch to bring the TAP test docs for PostgresNode up to date.

Peter Eisentraut sent in another revision of a patch to generate fmgr prototypes automatically.

Masahiko Sawada sent in a patch to add a GUC for cleanup indexes threshold.

Ashutosh Bapat sent in a patch to add an option to EXPLAIN: SUMMARY option which behaves as: SUMMARY when ON prints planning time, and prints planning time in EXPLAIN EXECUTE output.

Takayuki Tsunakawa sent in two more revisions of a patch to support huge pages on Windows.

Haribabu Kommi sent in another revision of a patch to add macaddr 64 bit (EUI-64) datatype support.

Ashutosh Sharma sent in two more revisions of a patch to add microvacuum support for hash indexes.

Jesper Pedersen sent in another revision of a patch to add support for hash index in pageinspect contrib module.

Tatsuo Ishii sent in a patch to fix a mistaken tag in the documentation for user name maps.

Vitaly Burovoy sent in another revision of a patch to add check for overflow to 'interval' functions.

Andrew Gierth sent in a patch to add hash support for grouping sets.

Jim Nasby sent in another revision of a patch to implement faster methods for getting SPI results in PL/PythonU.

Vladimir Rusinov sent in another revision of a patch to rename things xlog to things wal.

Jonathon Nelson sent in a patch to guc-ify the formerly hard-coded MAX_SEND_SIZE to max_wal_send.

Takeshi Ideriha sent in another revision of a patch to add DECLARE STATEMENT setting up a connection in ECPG.

Amit Langote sent in a patch to fix up some infelicities in declarative partitioning.

Ashutosh Sharma sent in another revision of a patch to add pgstathashindex() to get hash index table statistics.

Thomas Munro sent in another revision of a patch to implement barriers.

Thomas Munro sent in another revision of a patch to add parallel shared hash.

Peter Geoghegan sent in a patch to fix a subtle bug in "Simplify tape block format" commit.

Masahiko Sawada sent in another revision of a patch to enable block level parallel vacuum.

Michaël Paquier sent in another revision of a patch to add compression levels to pg_receivexlog.

Amul Sul sent in two more revisions of a patch to add a pg_background contrib module.

Pavel Stěhule sent in a patch to add pragmas to PL/pgsql.

Fabien COELHO sent in another revision of a patch to enable pgbench to store select results into variables.

Tom Lane sent in a patch to disallow output columns of type UNKNOWN.

Ryan Murphy sent in a patch to show type name and constraints for errors in inherited tables.

Jim Nasby sent in two more revisions of a patch to add more test coverage for PL/Tcl.

Victor Wagner sent in two revisions of a patch to add explicit subtransactions to PL/Tcl.

Haribabu Kommi sent in a patch to fix a bug in the patch to add a pg_hba_settings view.

par N Bougain le dimanche 15 janvier 2017 à 10h46

samedi 7 janvier 2017

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 1er janvier 2017

Les nouveautés des produits dérivés

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20170101214521.GA22193@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Remove triggerable Assert in hashname(). hashname() asserted that the key string it is given is shorter than NAMEDATALEN. That should surely always be true if the input is in fact a regular value of type "name". However, for reasons of coding convenience, we allow plain old C strings to be treated as "name" values in many places. Some SQL functions accept arbitrary "text" inputs, convert them to C strings, and pass them otherwise-untransformed to syscache lookups for name columns, allowing an overlength input value to trigger hashname's Assert. This would be a DOS problem, except that it only happens in assert-enabled builds which aren't recommended for production. In a production build, you'll just get a name lookup error, since regardless of the hash value computed by hashname, the later equality comparison checks can't match. Likewise, if the catalog lookup is done by seqscan or indexscan searches, there will just be a lookup error, since the name comparison functions don't contain any similar length checks, and will see an overlength input as unequal to any stored entry. After discussion we concluded that we should simply remove this Assert. It's inessential to hashname's own functionality, and having such an assertion in only some paths for name lookup is more of a foot-gun than a useful check. There may or may not be a case for the affected callers to do something other than let the name lookup fail, but we'll consider that separately; in any case we probably don't want to change such behavior in the back branches. Per report from Tushar Ahuja. Back-patch to all supported branches. Report: https://postgr.es/m/7d0809ee-6f25-c9d6-8e74-5b2967830d49@enterprisedb.com Discussion: https://postgr.es/m/17691.1482523168@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/54386f3578258caa5a1de97c434eee2c9ee2ab06
  • Fix interval_transform so it doesn't throw away non-no-op casts. interval_transform() contained two separate bugs that caused it to sometimes mistakenly decide that a cast from interval to restricted interval is a no-op and throw it away. First, it was wrong to rely on dt.h's field type macros to have an ordering consistent with the field's significance; in one case they do not. This led to mistakenly treating YEAR as less significant than MONTH, so that a cast from INTERVAL MONTH to INTERVAL YEAR was incorrectly discarded. Second, fls(1<<k) produces k+1 not k, so comparing its output directly to SECOND was wrong. This led to supposing that a cast to INTERVAL MINUTE was really a cast to INTERVAL SECOND and so could be discarded. To fix, get rid of the use of fls(), and make a function based on intervaltypmodout to produce a field ID code adapted to the need here. Per bug #14479 from Piotr Stefaniak. Back-patch to 9.2 where transform functions were introduced, because this code was born broken. Discussion: https://postgr.es/m/20161227172307.10135.7747@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/f0774abde868e0b5a2acbe75b5028884752f739d
  • Remove manual breaks in NodeTag assignments to fix duplicate tag numbers. Commit f0e44751d added new node tags at a place in the tag numbering where there was no daylight left before the next hard-coded number, resulting in some duplicate tag assignments. This doesn't seem to have caused any big problem so far, but it's surely trouble waiting to happen. We could adjust the manually assigned breakpoints to make more room, but that just leaves the same hazard waiting to strike again in future. What seems like a better idea is to get rid of the manual assignments and leave NodeTags to be automatically assigned, consecutively from one on up. This means that any change in the tag list forces a backend-wide recompile, but realistically that's usually needed anyway. Discussion: https://postgr.es/m/29670.1482942811@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/80a7298b9eb7f108ef20be6ee00d9513a43c61a0
  • Fix unstable regression test results. Commit 2ac3ef7a0 added a query with an underdetermined output row order; it has failed multiple times in the buildfarm since then. Add an ORDER BY to fix. Also, don't rely on a DROP CASCADE to drop in a well-determined order; that hasn't failed yet but I don't trust it much, and we're not saving any typing by using CASCADE anyway. http://git.postgresql.org/pg/commitdiff/257d8157205a7be5f9799e8941b922521d678a25
  • Avoid assuming that instr_time == struct timeval in pgbench logging. This code was presuming undue familiarity with the contents of the instr_time struct. That was already broken on Windows, and it's about to get broken on most other platforms as well. The simplest solution that preserves the current output definition is to just do our own gettimeofday() call here. Realistically, the extra cost is probably negligible in comparison to everything else that's going on in a pgbench transaction, so it's not worth sweating over. On Windows, the precision delivered by gettimeofday() is lower than one could wish, but this is still a big improvement over printing zeroes, as the code did before. Discussion: https://postgr.es/m/8837.1483216839@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/74baa1e3b89c1651ade1afeffc715cac24041e2f

Magnus Hagander pushed:

Andrew Dunstan pushed:

Peter Eisentraut pushed:

Correctifs en attente

Pavan Deolasee sent in another revision of a patch to fix some issues in WARM.

Kyotaro HORIGUCHI sent in another revision of a patch to clean up negative cache of pg_statistic when dropping a relation and clean up negative cache of pg_class when dropping a schema.

Kyotaro HORIGUCHI sent in another revision of a patch to redesign psql's tab completion and use that infrastructure to implement IF (NOT) EXISTS completion.

Ashutosh Bapat sent in another revision of a patch to imlement partition-wise join for join between (declaratively) partitioned tables.

Amit Kapila sent in another revision of a patch to implement WAL for hash indexes.

Dmitry Dolgov sent in another revision of a patch to implement generic type subscripting.

Amit Langote sent in five more revisions of a patch to implement declarative partitioning.

Amit Langote sent in a patch to fix an infelicity between OID columns and table inheritance.

Mithun Cy sent in two more revisions of a patch to cache hash index meta pages.

Etsuro Fujita sent in another revision of a patch to push more FULL joins to the PostgreSQL FDW.

Amit Kapila sent in another revision of a patch to implement parallel index scans.

Nikita Glukhov sent in another revision of a patch to implement recursive json_populate_record().

Peter Eisentraut sent in another revision of a patch to add ICU support.

Rafia Sabih sent in another revision of a patch to implement parallel index-only scans.

Amit Kapila sent in a patch to parallelize queries containing subplans.

Thomas Munro sent in a patch to fix dsa tranche registration.

Michaël Paquier sent in two more revisions of a patch to enable pg_receivelog to enable adjusting compression for tar format.

Peter Eisentraut sent in a patch to add some information about the systemd RemoveIPC issue to the documentation.

Ideriha Takeshi sent in a patch to add a DECLARE STATEMENT command in ECPG.

Craig Ringer sent in another revision of a patch to introduce txid_status(bigint) to get status of an xact.

Thomas Munro sent in another revision of a patch to allow measuring replay lag.

Claudio Freire sent in another revision of a patch to enable VACUUM to use more than 1GB of work mem.

Craig Ringer sent in two revisions of a patch to fix a minor race between commit_ts slru truncation and lookups.

Tom Lane sent in another revision of a patch to improve RLS planning.

Ãlvaro Herrera sent in two revisions of a patch to rewrite HeapSatisfiesHOTAndKey.

Ãlvaro Herrera sent in two more revisions of a patch to implement indirect indexes.

Vik Fearing sent in a patch to change the encoding of the French translation encoding from ISO-8859-1 to UTF-8.

Jim Nasby sent in another revision of a patch to increase pltcl test coverage.

Peter Eisentraut sent in another revision of a patch to add background sessions.

Tom Lane sent in a patch to eliminate gettimeofday() on platforms where better options are available.

Haribabu Kommi sent in a patch to implement a columnar storage extension.

Fabien COELHO sent in another revision of a patch to fix some pg_stat_statements query normalization issues with combined queries.

Yury Zhuravlev sent in another revision of a patch to implement a CMake-based build system.

Petr Jelínek sent in another revision of a patch to implement logical replication.

Michaël Paquier sent in another revision of a patch to fix a potential data loss of 2PC files.

Fabrízio de Royes Mello sent in another revision of a patch to implement COMMENT ON CURRENT DATABASE.

Amit Kapila sent in another revision of a patch to speed up clog Access by increasing CLOG buffers.

Amit Kapila sent in a patch to fix an issue where group clear xid can leak semaphore count.

Peter Eisentraut sent in a patch to generate fmgr prototypes automatically.

Peter Eisentraut sent in another revision of a patch to implement sequence data types.

Peter Eisentraut sent in a patch to put "use strict" in all Perl programs.

Peter Eisentraut sent in a patch to create the INSTALL file via XSLT and Pandoc. This is one more step on the way to replacing the old tool chain of jade and lynx.

Peter Eisentraut sent in a patch to automatically casts the result of copyNode() back to the input type, if the compiler supports something like typeof().

Stas Kelvich sent in another revision of a patch to implement logical decoding of two-phase transactions.

Magnus Hagander sent in another revision of a patch to support huge pages on Windows.

Peter Eisentraut sent in another revision of a patch to add identity columns per the SQL standard.

Magnus Hagander sent in a patch to change the backup and replication defaults to something more useful.

Peter Eisentraut sent in another revision of a patch to allow DROP FUNCTION to drop multiple functions in one command.

Peter Eisentraut sent in a patch to implement safer node casting.

Magnus Hagander sent in another revision of a patch to make streaming the pg_basebackup default.

David Fetter sent in another revision of a patch to add a hook which allows disabling simple UPDATEs and DELETEs without a WHERE clause.

Thomas Munro sent in a patch to add an isolation test for SERIALIZABLE READ ONLY DEFERRABLE.

Pavel Stěhule sent in a patch to add some runtime checks to PL/pgsql.

Simon Riggs sent in another revision of a patch to make some changes to the recovery.conf API.

par N Bougain le samedi 7 janvier 2017 à 12h46

Nouvelles hebdomadaires de PostgreSQL - 25 décembre 2016

Le pgDay Paris 2017 aura lieu à Paris (France) le 23 mars 2017. L'appel à conférenciers court jusqu'au 9 janvier : http://2017.pgday.paris/callforpapers/

Le PGDay nordique se tiendra à Stockholm (Suède) au Sheraton Hotel, le 21 mars 2017. L'appel à conférenciers expire le 9 janvier 2017 : https://2017.nordicpgday.org/cfp/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

  • La PGConf India 2017 aura lieu les 2 & 3 mars 2017 à Bengalore (État du Karnataka en Inde). Les propositions sont attendues sur <papers AT pgconf DOT in> jusqu'au 31 décembre 2016 : http://pgconf.in/
  • PostgreSQL@SCaLE aura lieu les 2 & 3 mars 2017 au centre des conventions de Pasadena, comme partie du SCaLE 15X : http://www.socallinuxexpo.org/scale/15x/
  • PgConf.Russia 2017 se déroulera du 15 au 17 mars 2017 à Moscou : https://pgconf.ru/en
  • Le PGDay Asia 2017 s'étendra du 17 au 18 mars à Singapour. L'appel à conférenciers s'éteindra le 16 janvier 2017 : http://tinyurl.com/pgDay-Asia-2017-Cfp
  • PGCon 2017 aura lieu à Ottawa du 23 au 26 mai. Les propositions seront attendues jusqu'au 19 janvier 2017 : http://www.pgcon.org/2017/papers.php

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161225225258.GA21777@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Magnus Hagander pushed:

Fujii Masao pushed:

  • Support quorum-based synchronous replication. This feature is also known as "quorum commit" especially in discussion on pgsql-hackers. This commit adds the following new syntaxes into synchronous_standby_names GUC. By using FIRST and ANY keywords, users can specify the method to choose synchronous standbys from the listed servers. FIRST num_sync (standby_name [, ...]) ANY num_sync (standby_name [, ...]) The keyword FIRST specifies a priority-based synchronous replication which was available also in 9.6 or before. This method makes transaction commits wait until their WAL records are replicated to num_sync synchronous standbys chosen based on their priorities. The keyword ANY specifies a quorum-based synchronous replication and makes transaction commits wait until their WAL records are replicated to *at least* num_sync listed standbys. In this method, the values of sync_state.pg_stat_replication for the listed standbys are reported as "quorum". The priority is still assigned to each standby, but not used in this method. The existing syntaxes having neither FIRST nor ANY keyword are still supported. They are the same as new syntax with FIRST keyword, i.e., a priorirty-based synchronous replication. Author: Masahiko Sawada Reviewed-By: Michael Paquier, Amit Kapila and me Discussion: <CAD21AoAACi9NeC_ecm+Vahm+MMA6nYh=Kqs3KB3np+MBOS_gZg@mail.gmail.com> Many thanks to the various individuals who were involved in discussing and developing this feature. http://git.postgresql.org/pg/commitdiff/3901fd70cc7ccacef1b0549a6835bb7d8dcaae43
  • Forbid invalid combination of options in pg_basebackup. Commit 56c7d8d4552180fd66fe48423bb2a9bb767c2d87 allowed pg_basebackup to stream WAL in tar mode. But there is the restriction that WAL streaming in tar mode works only when the value - (dash) is not specified as output directory. This means that the combination of three options "-D -", "-F t" and "-X stream" is invalid. However, previously, even when those options were specified at the same time, pg_basebackup background process unexpectedly started streaming WAL. And then it exited with an error. This commit changes pg_basebackup so that it errors out on such invalid combination of options at the beginning. Reviewed by Magnus Hagander, and patch by me. http://git.postgresql.org/pg/commitdiff/ecbdc4c555f43b1ac284c734752b00c2ea6f277b

Robert Haas pushed:

Tom Lane pushed:

  • Fix handling of phrase operator removal while removing tsquery stopwords. The distance of a removed phrase operator should propagate up to a parent phrase operator if there is one, but this only worked correctly in left-deep trees. Throwing in a few parentheses confused it completely, as indeed was illustrated by bizarre results in existing regression test cases. To fix, track unaccounted-for distances that should propagate to the left and to the right of the current node, rather than trying to make it work with only one returned distance. Also make some adjustments to behave as well as we can for cases of intermixed phrase and regular (AND/OR) operators. I don't think it's possible to be 100% correct for that without a rethinking of the tsquery representation; for example, maybe we should just not drop stopword nodes at all underneath phrase operators. But this is better than it was, and changing tsquery representation wouldn't be safely back-patchable. While at it, I simplified the API of the clean_fakeval_intree function a bit by getting rid of the "char *result" output parameter; that wasn't doing anything that wasn't redundant with whether the result node is NULL or not, and testing for NULL seems a lot clearer/safer. This is part of a larger project to fix various infelicities in the phrase-search implementation, but this part seems comittable on its own. Back-patch to 9.6 where phrase operators were introduced. Discussion: https://postgr.es/m/28215.1481999808@sss.pgh.pa.us Discussion: https://postgr.es/m/26706.1482087250@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/2604438472c897fbbd1568b1a8ee177ba8cdb6e3
  • Fix minor error message style violation. Primary error messages should not end with a period, since they're generally not written as full sentences. Oversight in 41493bac3. http://git.postgresql.org/pg/commitdiff/7d41a2bd3eef4de64ae8f6f683457f12f9407c5d
  • Fix minor oversights in nodeAgg.c. aggstate->evalproj is always set up by ExecInitAgg, so there's no need to test. Doing so led Coverity to think that we might be intending "slot" to be possibly NULL here, and it quite properly complained that the rest of combine_aggregates() wasn't prepared for that. Also fix a couple of obvious thinkos in Asserts checking that "inputoff" isn't past the end of the slot. Errors introduced in commit 8ed3f11bb, so no need for back-patch. http://git.postgresql.org/pg/commitdiff/c080b223a7a3991524a5287416a0ad756c15a098
  • Fix strange behavior (and possible crashes) in full text phrase search. In an attempt to simplify the tsquery matching engine, the original phrase search patch invented rewrite rules that would rearrange a tsquery so that no AND/OR/NOT operator appeared below a PHRASE operator. But this approach had numerous problems. The rearrangement step was missed by ts_rewrite (and perhaps other places), allowing tsqueries to be created that would cause Assert failures or perhaps crashes at execution, as reported by Andreas Seltenreich. The rewrite rules effectively defined semantics for operators underneath PHRASE that were buggy, or at least unintuitive. And because rewriting was done in tsqueryin() rather than at execution, the rearrangement was user-visible, which is not very desirable --- for example, it might cause unexpected matches or failures to match in ts_rewrite. As a somewhat independent problem, the behavior of nested PHRASE operators was only sane for left-deep trees; queries like "x <-> (y <-> z)" did not behave intuitively at all. To fix, get rid of the rewrite logic altogether, and instead teach the tsquery execution engine to manage AND/OR/NOT below a PHRASE operator by explicitly computing the match location(s) and match widths for these operators. This requires introducing some additional fields into the publicly visible ExecPhraseData struct; but since there's no way for third-party code to pass such a struct to TS_phrase_execute, it shouldn't create an ABI problem as long as we don't move the offsets of the existing fields. Another related problem was that index searches supposed that "!x <-> y" could be lossily approximated as "!x & y", which isn't correct because the latter will reject, say, "x q y" which the query itself accepts. This required some tweaking in TS_execute_ternary along with the main tsquery engine. Back-patch to 9.6 where phrase operators were introduced. While this could be argued to change behavior more than we'd like in a stable branch, we have to do something about the crash hazards and index-vs-seqscan inconsistency, and it doesn't seem desirable to let the unintuitive behaviors induced by the rewriting implementation stand as precedent. Discussion: https://postgr.es/m/28215.1481999808@sss.pgh.pa.us Discussion: https://postgr.es/m/26706.1482087250@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/89fcea1ace40bc025beea2758a80bcd56a319a6f
  • Fix detection of unfinished Unicode surrogate pair at end of string. The U&'...' and U&"..." syntaxes silently discarded a surrogate pair start (that is, a code between U+D800 and U+DBFF) if it occurred at the very end of the string. This seems like an obvious oversight, since we throw an error for every other invalid combination of surrogate characters, including the very same situation in E'...' syntax. This has been wrong since the pair processing was added (in 9.0), so back-patch to all supported branches. Discussion: https://postgr.es/m/19113.1482337898@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/a8ae12322ae056ebbe9d83cf16b4c92e86a6ac28
  • Give a useful error message if uuid-ossp is built without preconfiguration. Before commit b8cc8f947, it was possible to build contrib/uuid-ossp without having told configure you meant to; you could just cd into that directory and "make". That no longer works because the code depends on configure to have done header and library probes, but the ensuing error messages are not so easy to interpret if you're not an old C hand. We've gotten a couple of complaints recently from people trying to do this the low-tech way, so add an explicit #error directing the user to use --with-uuid. (In principle we might want to do something similar in the other optionally-built contrib modules; but I don't think any of the others have ever worked without preconfiguration, so there are no bad habits to break people of.) Back-patch to 9.4 where the previous commit came in. Report: https://postgr.es/m/CAHeEsBf42AWTnk=1qJvFv+mYgRFm07Knsfuc86Ono8nRjf3tvQ@mail.gmail.com Report: https://postgr.es/m/CAKYdkBrUaZX+F6KpmzoHqMtiUqCtAW_w6Dgvr6F0WTiopuGxow@mail.gmail.com http://git.postgresql.org/pg/commitdiff/b86515da1a73d0a2e23aca728f18b5f9e809e89f
  • Fix handling of expanded objects in CoerceToDomain and CASE execution. When the input value to a CoerceToDomain expression node is a read-write expanded datum, we should pass a read-only pointer to any domain CHECK expressions and then return the original read-write pointer as the expression result. Previously we were blindly passing the same pointer to all the consumers of the value, making it possible for a function in CHECK to modify or even delete the expanded value. (Since a plpgsql function will absorb a passed-in read-write expanded array as a local variable value, it will in fact delete the value on exit.) A similar hazard of passing the same read-write pointer to multiple consumers exists in domain_check() and in ExecEvalCase, so fix those too. The fix requires adding MakeExpandedObjectReadOnly calls at the appropriate places, which is simple enough except that we need to get the data type's typlen from somewhere. For the domain cases, solve this by redefining DomainConstraintRef.tcache as okay for callers to access; there wasn't any reason for the original convention against that, other than not wanting the API of typcache.c to be any wider than it had to be. For CASE, there's no good solution except to add a syscache lookup during executor start. Per bug #14472 from Marcos Castedo. Back-patch to 9.5 where expanded values were introduced. Discussion: https://postgr.es/m/15225.1482431619@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/cd1b215692d2cadf499936dba05f1f60bce59344
  • Fix CREATE TABLE ... LIKE ... WITH OIDS. Having a WITH OIDS specification should result in the creation of an OID column, but commit b943f502b broke that in the case that there were LIKE tables without OIDS. Commentary in that patch makes it look like this was intentional, but if so it was based on a faulty reading of what inheritance does: the parent tables can add an OID column, but they can't subtract one. AFAICS, the behavior ought to be that you get an OID column if any of the inherited tables, LIKE tables, or WITH clause ask for one. Also, revert that patch's unnecessary split of transformCreateStmt's loop over the tableElts list into two passes. That seems to have been based on a misunderstanding as well: we already have two-pass processing here, we don't need three passes. Per bug #14474 from Jeff Dafoe. Back-patch to 9.6 where the misbehavior was introduced. Report: https://postgr.es/m/20161222145304.25620.47445@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/1ead0208b2178089b024caa2d1465a3f3056a45c
  • Spellcheck: s/descendent/descendant/g. I got a little annoyed by reading documentation paragraphs containing both spellings within a few lines of each other. My dictionary says "descendant" is the preferred spelling, and it's certainly the majority usage in our tree, so standardize on that. For one usage in parallel.sgml, I thought it better to rewrite to avoid the term altogether. http://git.postgresql.org/pg/commitdiff/ff33d1456ea098e160cbbc74b332656c06abc2ab
  • Doc: improve index entry for "median". We had an index entry for "median" attached to the percentile_cont function entry, which was pretty useless because a person following the link would never realize that that function was the one they were being hinted to use. Instead, make the index entry point at the example in syntax-aggregates, and add a <seealso> link to "percentile". Also, since that example explicitly claims to be calculating the median, make it use percentile_cont not percentile_disc. This makes no difference in terms of the larger goals of that section, but so far as I can find, nearly everyone thinks that "median" means the continuous not discrete calculation. Per gripe from Steven Winfield. Back-patch to 9.4 where we introduced percentile_cont. Discussion: https://postgr.es/m/20161223102056.25614.1166@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/3c9d398484fb6e188e665be8299d6e5e89924c94
  • Replace enum InhOption with simple boolean. Now that it has only INH_NO and INH_YES values, it's just weird that it's not a plain bool, so make it that way. Also rename RangeVar.inhOpt to "inh", to be like RangeTblEntry.inh. My recollection is that we gave it a different name specifically because it had a different representation than the derived bool value, but it no longer does. And this is a good forcing function to be sure we catch any places that are affected by the change. Bump catversion because of possible effect on stored RangeVar nodes. I'm not exactly convinced that we ever store RangeVar on disk, but we have a readfuncs function for it, so be cautious. (If we do do so, then commit e13486eba was in error not to bump catversion.) Follow-on to commit e13486eba. Discussion: http://postgr.es/m/CA+TgmoYe+EG7LdYX6pkcNxr4ygkP4+A=jm9o-CPXyOvRiCNwaQ@mail.gmail.com http://git.postgresql.org/pg/commitdiff/fe591f8bf68db9bf81f278acce6239ee68cd4ed6
  • Fix incorrect error reporting for duplicate data in \crosstabview. \crosstabview's complaint about multiple entries for the same crosstab cell quoted the wrong row and/or column values. It would accidentally appear to work if the data had been in strcmp() order to start with, which probably explains how we missed noticing this during development. This could be fixed in more than one way, but the way I chose was to hang onto both result pointers from bsearch() and use those to get at the value names. In passing, avoid casting away const in the bsearch comparison functions. No bug there, just poor style. Per bug #14476 from Tomonari Katsumata. Back-patch to 9.6 where \crosstabview was introduced. Report: https://postgr.es/m/20161225021519.10139.45460@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/a3aef88e6a9c5822eb4a5ad0744b15dc6e8a5d86

Heikki Linnakangas pushed:

Peter Eisentraut pushed:

Dean Rasheed pushed:

  • Fix order of operations in CREATE OR REPLACE VIEW. When CREATE OR REPLACE VIEW acts on an existing view, don't update the view options until after the view query has been updated. This is necessary in the case where CREATE OR REPLACE VIEW is used on an existing view that is not updatable, and the new view is updatable and specifies the WITH CHECK OPTION. In this case, attempting to apply the new options to the view before updating its query fails, because the options are applied using the ALTER TABLE infrastructure which checks that WITH CHECK OPTION is only applied to an updatable view. If new columns are being added to the view, that is also done using the ALTER TABLE infrastructure, but it is important that that still be done before updating the view query, because the rules system checks that the query columns match those on the view relation. Added a comment to explain that, in case someone is tempted to move that to where the view options are now being set. Back-patch to 9.4 where WITH CHECK OPTION was added. Report: https://postgr.es/m/CAEZATCUp%3Dz%3Ds4SzZjr14bfct_bdJNwMPi-gFi3Xc5k1ntbsAgQ%40mail.gmail.com http://git.postgresql.org/pg/commitdiff/58b1362642d47bd7a7ed1157035a38671555e860

Stephen Frost pushed:

  • Fix dumping of casts and transforms using built-in functions. In pg_dump.c dumpCast() and dumpTransform(), we would happily ignore the cast or transform if it happened to use a built-in function because we weren't including the information about built-in functions when querying pg_proc from getFuncs(). Modify the query in getFuncs() to also gather information about functions which are used by user-defined casts and transforms (where "user-defined" means "has an OID >= FirstNormalObjectId"). This also adds to the TAP regression tests for 9.6 and master to cover these types of objects. Back-patch all the way for casts, back to 9.5 for transforms. Discussion: https://www.postgresql.org/message-id/flat/20160504183952.GE10850%40tamriel.snowman.net http://git.postgresql.org/pg/commitdiff/2259bf672cb45b4104dcb835354beeb1c6105b0e
  • For 8.0 servers, get last built-in oid from pg_database. We didn't start ensuring that all built-in objects had OIDs less than 16384 until 8.1, so for 8.0 servers we still need to query the value out of pg_database. We need this, in particular, to distinguish which casts were built-in and which were user-defined. For HEAD, we only worry about going back to 8.0, for the back-branches, we also ensure that 7.0-7.4 work. Discussion: https://www.postgresql.org/message-id/flat/20160504183952.GE10850%40tamriel.snowman.net http://git.postgresql.org/pg/commitdiff/19990918d3fa2a445561627ed415b5891602f7fe
  • Improve ALTER TABLE documentation. The ALTER TABLE documentation wasn't terribly clear when it came to which commands could be combined together and what it meant when they were. In particular, SET TABLESPACE *can* be combined with other commands, when it's operating against a single table, but not when multiple tables are being moved with ALL IN TABLESPACE. Further, the actions are applied together but not really in 'parallel', at least today. Pointed out by: Amit Langote. Improved wording from Tom. Back-patch to 9.4, where the ALL IN TABLESPACE option was added. Discussion: https://www.postgresql.org/message-id/14c535b4-13ef-0590-1b98-76af355a0763%40lab.ntt.co.jp http://git.postgresql.org/pg/commitdiff/2d1018ca56f5ddaf0bfb5b4d0133283f3e823301
  • Use TSConfigRelationId in AlterTSConfiguration(). When we are altering a text search configuration, we are getting the tuple from pg_ts_config and using its OID, so use TSConfigRelationId when invoking any post-alter hooks and setting the object address. Further, in the functions called from AlterTSConfiguration(), we're saving information about the command via EventTriggerCollectAlterTSConfig(), so we should be setting commandCollected to true. Also add a regression test to test_ddl_deparse for ALTER TEXT SEARCH CONFIGURATION. Author: Artur Zakirov, a few additional comments by me Discussion: https://www.postgresql.org/message-id/57a71eba-f2c7-e7fd-6fc0-2126ec0b39bd%40postgrespro.ru Back-patch the fix for the InvokeObjectPostAlterHook() call to 9.3 where it was introduced, and the fix for the ObjectAddressSet() call and setting commandCollected to true to 9.5 where those changes to ProcessUtilitySlow() were introduced. http://git.postgresql.org/pg/commitdiff/12bd7dd317e8f4346fb3507578aca790ede6ebea
  • Fix tab completion in psql for ALTER DEFAULT PRIVILEGES. When providing tab completion for ALTER DEFAULT PRIVILEGES, we are including the list of roles as possible options for completion after the GRANT or REVOKE. Further, we accept FOR ROLE/IN SCHEMA at the same time and in either order, but the tab completion was only working for one or the other. Lastly, we weren't using the actual list of allowed kinds of objects for default privileges for completion after the 'GRANT X ON' but instead were completeing to what 'GRANT X ON' supports, which isn't the ssame at all. Address these issues by improving the forward tab-completion for ALTER DEFAULT PRIVILEGES and then constrain and correct how the tail completion is done when it is for ALTER DEFAULT PRIVILEGES. Back-patch the forward/tail tab-completion to 9.6, where we made it easy to handle such cases. For 9.5 and earlier, correct the initial tab-completion to at least be correct as far as it goes and then add a check for GRANT/REVOKE to only tab-complete when the GRANT/REVOKE is the start of the command, so we don't try to do tab-completion after we get to the GRANT/REVOKE part of the ALTER DEFAULT PRIVILEGES command, which is better than providing incorrect completions. Initial patch for master and 9.6 by Gilles Darold, though I cleaned it up and added a few comments. All bugs in the 9.5 and earlier patch are mine. Discussion: https://www.postgresql.org/message-id/1614593c-e356-5b27-6dba-66320a9bc68b@dalibo.com http://git.postgresql.org/pg/commitdiff/f3fd531a51df2a73d8517a542e6999e0186e586b
  • pg_dumpall: Include --verbose option in --help output. The -v/--verbose option was not included in the output from --help for pg_dumpall even though it's in the pg_dumpall documentation and has apparently been around since pg_dumpall was reimplemented in C in 2002. Fix that by adding it. Pointed out by Daniel Westermann. Back-patch to all supported branches. Discussion: https://www.postgresql.org/message-id/2020970042.4589542.1482482101585.JavaMail.zimbra%40dbi-services.com http://git.postgresql.org/pg/commitdiff/86d216c77549e200b95bed487b6fb87d99a1e789

Joe Conway pushed:

  • Improve dblink error message when remote does not provide it. When dblink or postgres_fdw detects an error on the remote side of the connection, it will try to construct a local error message as best it can using libpq's PQresultErrorField(). When no primary message is available, it was bailing out with an unhelpful "unknown error". Make that message better and more style guide compliant. Per discussion on hackers. Backpatch to 9.2 except postgres_fdw which didn't exist before 9.3. Discussion: https://postgr.es/m/19872.1482338965%40sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/ea0aa9698cfa74bb04cf53d813924fe67f278c30
  • Protect dblink from invalid options when using postgres_fdw server. When dblink uses a postgres_fdw server name for its connection, it is possible for the connection to have options that are invalid with dblink (e.g. "updatable"). The recommended way to avoid this problem is to use dblink_fdw servers instead. However there are use cases for using postgres_fdw, and possibly other FDWs, for dblink connection options, therefore protect against trying to use any options that do not apply by using is_valid_dblink_option() when building the connection string from the options. Back-patch to 9.3. Although 9.2 supports FDWs for connection info, is_valid_dblink_option() did not yet exist, and neither did postgres_fdw, at least in the postgres source tree. Given the lack of previous complaints, fixing that seems too invasive/not worth it. Author: Corey Huinker Reviewed-By: Joe Conway Discussion: https://postgr.es/m/CADkLM%3DfWyXVEyYcqbcRnxcHutkP45UHU9WD7XpdZaMfe7S%3DRwA%40mail.gmail.com http://git.postgresql.org/pg/commitdiff/c4448683893bd37b59003603bc9075d362e81b5a
  • Make dblink try harder to form useful error messages. When libpq encounters a connection-level error, e.g. runs out of memory while forming a result, there will be no error associated with PGresult, but a message will be placed into PGconn's error buffer. postgres_fdw takes care to use the PGconn error message when PGresult does not have one, but dblink has been negligent in that regard. Modify dblink to mirror what postgres_fdw has been doing. Back-patch to all supported branches. Author: Joe Conway Reviewed-By: Tom Lane Discussion: https://postgr.es/m/02fa2d90-2efd-00bc-fefc-c23c00eb671e%40joeconway.com http://git.postgresql.org/pg/commitdiff/2f802d95b4904dbed3dfdca1b3a607cd085d2e20
  • Improve RLS documentation with respect to COPY. Documentation for pg_restore said COPY TO does not support row security when in fact it should say COPY FROM. Fix that. While at it, make it clear that "COPY FROM" does not allow RLS to be enabled and INSERT should be used instead. Also that SELECT policies will apply to COPY TO statements. Back-patch to 9.5 where RLS first appeared. Author: Joe Conway Reviewed-By: Dean Rasheed and Robert Haas Discussion: https://postgr.es/m/5744FA24.3030008%40joeconway.com http://git.postgresql.org/pg/commitdiff/0a85c102254b72ec7ce16bc504206a1a5c84bd76

Michael Meskes pushed:

Andres Freund pushed:

Correctifs en attente

Thomas Munro sent in a patch to add support for delta relations in AFTER triggers to PL/PythonU.

Petr Jelínek sent in four more revisions of a patch to add logical replication.

Petr Jelínek sent in a patch to enable logical replication of data which was there before logical replication was turned on.

Michaël Paquier sent in a patch to change detection of corrupted 2PC files as FATAL and minimize window between history file and end-of-recovery record.

Kyotaro HORIGUCHI sent in a patch to protect syscache from bloating with negative cache entries.

Thomas Munro sent in another revision of a patch to allow measuring replay lag.

Beena Emerson sent in a patch to make wal segment size initdb configurable.

Ashutosh Sharma sent in another revision of a patch to add support for hash index in pageinspect contrib module.

Michaël Paquier sent in two revisions of a patch to add tests for authentication and pg_hba.conf.

Heikki Linnakangas sent in a patch to use the "plain:" prefix for plaintext passwords stored in pg_authid.

Dilip Kumar sent in another revision of a patch to implement parallel bitmap heap scan.

Ants Aasma and Craig Ringer traded patches to fix an issue where the replication slot xmin is not reset if HS feedback is turned off while the standby is shut down.

Kyotaro HORIGUCHI sent in another revision of a patch to implement asynchronous query execution.

Heikki Linnakangas sent in another revision of a patch to add logical tape pause/resume.

Ashutosh Sharma sent in a patch to create a pgstathashindex() function to get hash index table statistics.

Dilip Kumar sent in another revision of a patch to implement parallel merge join.

Andrea Urbani sent in a patch to add --custom-fetch-table and --custom-fetch-value parameters to pg_dump.

Etsuro Fujita and Ashutosh Bapat traded patches to fix a FDW bug.

Artur Zakirov sent in another revision of a patch to output pg_ts_config_map entries with pg_event_trigger_ddl_commands().

Michaël Paquier sent in a patch to fix a potential data loss of 2PC files.

Ashutosh Bapat sent in a patch to translate partition hierarchy into inheritance hierarchy.

Craig Ringer sent in three revisions of a patch to introduce txid_status(bigint) to get status of an xact.

Pavel Stěhule sent in another revision of a patch to add xmltable().

Kuntal Ghosh sent in a patch to reduce WALWriteLock contention.

Amul Sul sent in another revision of a patch to add a pg_background contrib extension.

Amit Khandekar sent in a patch to implement parallel append.

Claudio Freire sent in a patch to prefetch buffers on backward scan and allow using more than 1GB work mem in VACUUM.

Ãlvaro Herrera sent in two more revisions of a patch to implement indirect indexes.

Craig Ringer sent in a patch to add a new CREATE_REPLICATION_SLOT option.

Daniel Vérité sent in another revision of a patch to improve some of the psql variable hooks.

Jerry Yu sent in a patch to push down quals into EXCEPT.

Fabien COELHO sent in two more revisions of a patch to fix the handling of compound/combined queries by pg_stat_statements.

Joel Jacobson sent in a patch to implement seconds resolution for wait_start.

Rafia Sabih sent in another revision of a patch to implement parallel index-only scans.

Erik Rijkers sent in a patch to fix a typo in a comment in tablecmds.c.

par N Bougain le samedi 7 janvier 2017 à 12h17

mardi 20 décembre 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 18 décembre 2016

[ndt: Meetup du PLUG à Lyon ce mardi 20 décembre : https://www.meetup.com/fr-FR/PostgreSQL-Lyon-User-Group/]

PGCon 2017 aura lieu à Ottawa du 23 au 26 mai. Les propositions seront attendues jusqu'au 19 janvier 2017 : http://www.pgcon.org/2017/papers.php

PgConf.Russia 2017 se déroulera du 15 au 17 mars 2017 à Moscou : https://pgconf.ru/en

Le PGDay Asia 2017 s'étendra du 17 au 18 mars à Singapour. L'appel à conférenciers s'éteindra le 16 janvier 2017 : http://tinyurl.com/pgDay-Asia-2017-Cfp

Les inscriptions pour le PGDay du FOSDEM 2017 à Bruxelles sont à présent ouvertes : https://2017.fosdempgday.org/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

  • La PGConf India 2017 aura lieu les 2 & 3 mars 2017 à Bengalore (État du Karnataka en Inde). Les propositions sont attendues sur <papers AT pgconf DOT in> jusqu'au 31 décembre 2016 : http://pgconf.in/
  • PostgreSQL@SCaLE aura lieu les 2 & 3 mars 2017 au centre des conventions de Pasadena, comme partie du SCaLE 15X : http://www.socallinuxexpo.org/scale/15x/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161219002906.GA8714@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Heikki Linnakangas pushed:

Robert Haas pushed:

  • psql: Fix incorrect version check for table partitioning. Table partitioning was added in 10, not 9.6. Fabrízio de Royes Mello, per report from Jeff Janes http://git.postgresql.org/pg/commitdiff/06e184876bc07c2b1d7144957dcf02c5b4f709c2
  • doc: Fix purported type of pg_am.amhandler to match reality. Joel Jacobson http://git.postgresql.org/pg/commitdiff/b4630e01fd4c73c195025b7307ebc13d489b9ef9
  • Remove should_free arguments to tuplesort routines. Since commit e94568ecc10f2638e542ae34f2990b821bbf90ac, the answer is always "false", and we do not need to complicate the API by arranging to return a constant value. Peter Geoghegan Discussion: http://postgr.es/m/CAM3SWZQWZZ_N=DmmL7tKy_OUjGH_5mN=N=A6h7kHyyDvEhg2DA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/3856cf9607f41245ec9462519c53f1109e781fc5
  • doc: Improve documentation related to table partitioning feature. Commit f0e44751d7175fa3394da2c8f85e3ceb3cdbfe63 implemented table partitioning, but failed to mention the "no row movement" restriction in the documentation. Fix that and a few other issues. Amit Langote, with some additional wordsmithing by me. http://git.postgresql.org/pg/commitdiff/a1a4459c299a86f909c27e391a10d7b9b05ea697
  • Update typedefs.list. So developers can more easily run pgindent locally http://git.postgresql.org/pg/commitdiff/acddbe221b084956a0efd6e4b6c6586e8fd994d7
  • Clean up code, comments, and formatting for table partitioning. Amit Langote, plus pgindent-ing by me. Inspired in part by review comments from Tomas Vondra. http://git.postgresql.org/pg/commitdiff/4b9a98e154cec81849af24091443747a6057c968
  • Fix bug in hashbulkdelete. Commit 6d46f4783efe457f74816a75173eb23ed8930020 failed to account for the possibility that hashbulkdelete() might encounter a bucket that has been split since it began scanning the bucket array. Repair. Extracted from a larger pathc by Amit Kapila; I rewrote the comment. http://git.postgresql.org/pg/commitdiff/501c7b94bcb00cfa0faad60135cf6af82fd56a3a
  • Fix bugs in RelationGetPartitionDispatchInfo. The previous coding was not quite right for cases involving multiple levels of partitioning. Amit Langote http://git.postgresql.org/pg/commitdiff/a25665088d812d08bb888e961f208eaebf522050
  • Remove _hash_wrtbuf() in favor of calling MarkBufferDirty(). The whole concept of _hash_wrtbuf() is that we need to know at the time we're releasing the buffer lock (and pin) whether we dirtied the buffer, but this is easy to get wrong. This patch actually fixes one non-obvious bug of that form: hashbucketcleanup forgot to signal _hash_squeezebucket, which gets the primary bucket page already locked, as to whether it had already dirtied the page. Calling MarkBufferDirty() at the places where we dirty the buffer is more intuitive and lets us simplify the code in various places as well. On top of all that, the ultimate goal here is to make hash indexes WAL-logged, and as the comments to _hash_wrtbuf() note, it should go away when that happens. Making it go away a little earlier than that seems like a good preparatory step. Report by Jeff Janes. Diagnosis by Amit Kapila, Kuntal Ghosh, and Dilip Kumar. Patch by me, after studying an alternative patch submitted by Amit Kapila. Discussion: http://postgr.es/m/CAA4eK1Kf6tOY0oVz_SEdngiNFkeXrA3xUSDPPORQvsWVPdKqnA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/25216c98938495fd741bf585dcbef45b3a9ffd40
  • Fix more hash index bugs around marking buffers dirty. In _hash_freeovflpage(), if we're freeing the overflow page that immediate follows the page to which tuples are being moved (the confusingly-named "write buffer"), don't forget to mark that page dirty after updating its hasho_nextblkno. In _hash_squeezebucket(), it's not necessary to mark the primary bucket page dirty if there are no overflow pages, because there's nothing to squeeze in that case. Amit Kapila, with help from Kuntal Ghosh and Dilip Kumar, after an initial trouble report by Jeff Janes. http://git.postgresql.org/pg/commitdiff/6a4fe1127c5a0ea1515589e416aa29e088170c0e
  • Unbreak Finalize HashAggregate over Partial HashAggregate. Commit 5dfc198146b49ce7ecc8a1fc9d5e171fb75f6ba5 introduced the use of a new type of hash table with linear reprobing for hash aggregates. Such a hash table behaves very poorly if keys are inserted in hash order, which does in fact happen in the case where a query use a Finalize HashAggregate node fed (via Gather) by a Partial HashAggregate node. In fact, queries with this type of plan tend to run effectively forever. Fix that by seeding the hash value differently in each worker (and in the leader, if it participates). Andres Freund and Robert Haas http://git.postgresql.org/pg/commitdiff/b81b5a96f424531b97cdd1dba97d9d1b9c9d372e
  • Simplify LWLock tranche machinery by removing array_base/array_stride. array_base and array_stride were added so that we could identify the offset of an LWLock within a tranche, but this facility is only very marginally used apart from the main tranche. So, give every lock in the main tranche its own tranche ID and get rid of array_base, array_stride, and all that's attached. For debugging facilities (Trace_lwlocks and LWLOCK_STATS) print the pointer address of the LWLock using %p instead of the offset. This is arguably more useful, and certainly a lot cheaper. Drop the offset-within-tranche from the information reported to dtrace and from one can't-happen message inside lwlock.c. The main user-visible impact of this change is that pg_stat_activity will now report all waits for LWLocks as "LWLock" rather than reporting some as "LWLockTranche" and others as "LWLockNamed". The main motivation for this change is that the need to specify an array_base and an array_stride is awkward for parallel query. There is only a very limited supply of tranche IDs so we can't just keep allocating new ones, and if we try to use the same tranche IDs every time then we run into trouble when multiple parallel contexts are use simultaneously. So if we didn't get rid of this mechanism we'd have to make it even more complicated. By simplifying it in this way, we instead reduce the size of the generated code for lwlock.c by about 5%. Discussion: http://postgr.es/m/CA+TgmoYsFn6NUW1x0AZtupJGUAs1UDY4dJtCN47_Q6D0sP80PA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/3761fe3c20bb040b15f0e8da58d824631da00caa
  • Fix outdated comment in lwlock.c Commit 3761fe3c20bb040b15f0e8da58d824631da00caa should have made this change, but didn't. Reported by Ãlvaro Herrera. http://git.postgresql.org/pg/commitdiff/591ccb66d24258f6d1084343b3c33c96e3e2b36d

Tom Lane pushed:

  • Make the different Unix-y semaphore implementations ABI-compatible. Previously, the "sem" field of PGPROC varied in size depending on which kernel semaphore API we were using. That was okay as long as there was only one likely choice per platform, but in the wake of commit ecb0d20a9, that assumption seems rather shaky. It doesn't seem out of the question anymore that an extension compiled against one API choice might be loaded into a postmaster built with another choice. Moreover, this prevents any possibility of selecting the semaphore API at postmaster startup, which might be something we want to do in future. Hence, change PGPROC.sem to be PGSemaphore (i.e. a pointer) for all Unix semaphore APIs, and turn the pointed-to data into an opaque struct whose contents are only known within the responsible modules. For the SysV and unnamed-POSIX APIs, the pointed-to data has to be allocated elsewhere in shared memory, which takes a little bit of rejiggering of the InitShmemAllocation code sequence. (I invented a ShmemAllocUnlocked() function to make that a little cleaner than it used to be. That function is not meant for any uses other than the ones it has now, but it beats having InitShmemAllocation() know explicitly about allocation of space for semaphores and spinlocks.) This change means an extra indirection to access the semaphore data, but since we only touch that when blocking or awakening a process, there shouldn't be any meaningful performance penalty. Moreover, at least for the unnamed-POSIX case on Linux, the sem_t type is quite a bit wider than a pointer, so this reduces sizeof(PGPROC) which seems like a good thing. For the named-POSIX API, there's effectively no change: the PGPROC.sem field was and still is a pointer to something returned by sem_open() in the postmaster's memory space. Document and check the pre-existing limitation that this case can't work in EXEC_BACKEND mode. It did not seem worth unifying the Windows semaphore ABI with the Unix cases, since there's no likelihood of needing ABI compatibility much less runtime switching across those cases. However, we can simplify the Windows code a bit if we define PGSemaphore as being directly a HANDLE, rather than pointer to HANDLE, so let's do that while we're here. (This also ends up being no change in what's physically stored in PGPROC.sem. We're just moving the HANDLE fetch from callees to callers.) It would take a bunch of additional code shuffling to get to the point of actually choosing a semaphore API at postmaster start, but the effects of that would now be localized in the port/XXX_sema.c files, so it seems like fit material for a separate patch. The need for it is unproven as yet, anyhow, whereas the ABI risk to extensions seems real enough. Discussion: https://postgr.es/m/4029.1481413370@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/be7b2848c6d8bdbfb63ab403c535713708c4af52
  • Fix race condition in test_decoding "slot" test. This test, just added in commit a924c327e, sometimes fails because the old backend hasn't finished dropping the temporary replication slot when the new backend looks. Borrow the previously-invented methodology for waiting for the old process to disappear from pg_stat_activity. Petr Jelinek Discussion: https://postgr.es/m/62935e6f-4f1b-c433-e0fa-7f936a38b3e5@2ndquadrant.com http://git.postgresql.org/pg/commitdiff/23f722ba8e19ca1a7c2ada9d6e687989b6e8f4d1
  • Catversion bump for temporary replication slots. Missed in commit a924c327e2793d2025b19e18de7917110dc8afd8. Per Fujii Masao. http://git.postgresql.org/pg/commitdiff/9b3d02c2a9eb93cc4754857361abee449a3fe0cb
  • Fix creative, but unportable, spelling of "ptr != NULL". Or at least I suppose that's what was really meant here. But even aside from the not-per-project-style use of "0" to mean "NULL", I doubt it's safe to assume that all valid pointers are > NULL. Per buildfarm member pademelon. http://git.postgresql.org/pg/commitdiff/563d575fd73361f6118c13f2816988eba8e1f657
  • Prevent planagg.c from failing on queries containing CTEs. The existing tests in preprocess_minmax_aggregates() usually prevent it from trying to do anything with queries containing CTEs, but there's an exception: a CTE could be present as a member of an appendrel, if we flattened a UNION ALL that contains CTE references. If it did try to generate an optimized path for a query using a CTE, it failed with "could not find plan for CTE", as reported by Torsten Förtsch. The proximate cause is an unwise decision in commit 3fc6e2d7f to clear subroot->cte_plan_ids in build_minmax_path(). That left the subroot's cte_plan_ids list out of step with its parse->cteList. Removing the "subroot->cte_plan_ids = NIL;" assignment is enough to let the case work again, but really it's pretty silly to be expending any cycles at all in this module when there are CTEs: we always treat their outputs as unordered so there's no way for the optimization to win. Hence, also add an early-exit test so we don't waste time like that. Back-patch to 9.6 where the misbehavior was introduced. Report: https://postgr.es/m/CAKkG4_=gjY5QiHtqSZyWMwDuTd_CftKoTaCqxjJ7uUz1-Gw=qw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/1f542a2eacca030c676cbb594f3b362d43f2f857
  • Improve handling of array elements as getdiag_targets and cursor_variables. There's no good reason why plpgsql's GET DIAGNOSTICS statement can't support an array element as target variable, since the execution code already uses the generic exec_assign_value() function to assign to it. Hence, refactor the grammar to allow that, by making getdiag_target depend on the assign_var production. Ideally we'd also let a cursor_variable expand to an element of a refcursor[] array, but that's substantially harder since those statements also have to handle bound-cursor-variable cases. For now, just make sure the reported error is sensible, ie "cursor variable must be a simple variable" not "variable must be of type cursor or refcursor". The latter was quite confusing from the user's viewpoint, since what he wrote satisfies the claimed restriction. Per bug #14463 from Zhou Digoal. Given the lack of previous complaints, I see no need for a back-patch. Discussion: https://postgr.es/m/20161213152548.14897.81245@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/55caaaeba877eac1feb6481fb413fa04ae9046ac
  • Sync our copy of the timezone library with IANA release tzcode2016j. This is a trivial update (consisting in fact only in the addition of a comment). The point is just to get back to being synced with an official release of tzcode, rather than some ad-hoc point in their commit history, which is where commit 1f87181e1 left it. http://git.postgresql.org/pg/commitdiff/93513d1b6559b2d0805f0b02d312ee550e3d010b
  • Improve documentation around TS_execute(). I got frustrated by the lack of commentary in this area, so here is some reverse-engineered documentation, along with minor stylistic cleanup. No code changes more significant than removal of unused variables. Back-patch to 9.6, not because that's useful in itself, but because we have some bugs to fix in phrase search and this would cause merge failures if it's only in HEAD. http://git.postgresql.org/pg/commitdiff/23c75b55aaccddea79545ffaf1cbfc9f1edeaa8c
  • Fix FK-based join selectivity estimation for semi/antijoins. This case wasn't thought through sufficiently in commit 100340e2d. It's true that the FK proves that every outer row has a match in the inner table, but we forgot that some of the inner rows might be filtered away by WHERE conditions located within the semijoin's RHS. If the RHS is just one table, we can reasonably take the semijoin selectivity as equal to the fraction of the referenced table's rows that are expected to survive its restriction clauses. If the RHS is a join, it's not clear how much of the referenced table might get through the join, so fall back to the same rule we were already using for other outer-join cases: use the minimum of the regular per-clause selectivity estimates. This gives the same result as if we hadn't considered the FK at all when there's a single FK column, but it should still help for multi-column FKs, which is the case that 100340e2d is really meant to help with. Back-patch to 9.6 where the previous commit came in. Discussion: https://postgr.es/m/16149.1481835103@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/7fa93eec4e0c9c3e801e3c51aa4bae3a38aaa218
  • In contrib/uuid-ossp, #include headers needed for ntohl() and ntohs(). Oversight in commit b8cc8f947. I just noticed this causes compiler warnings on FreeBSD, and it really ought to cause warnings elsewhere too: all references I can find say that <arpa/inet.h> is required for these. We have a lot of code elsewhere that thinks that both <netinet/in.h> and <arpa/inet.h> should be included for these functions, so do it that way here too, even though <arpa/inet.h> ought to be sufficient according to the references I consulted. Back-patch to 9.4 where the previous commit landed. http://git.postgresql.org/pg/commitdiff/4a0a34b5b678f0292d3a85a85fb10c79c393be26

Peter Eisentraut pushed:

Magnus Hagander pushed:

Fujii Masao pushed:

  • Add missing documentation for effective_io_concurrency tablespace option. The description of effective_io_concurrency option was missing in ALTER TABLESPACE docs though it's included in CREATE TABLESPACE one. Back-patch to 9.6 where effective_io_concurrency tablespace option was added. Michael Paquier, reported by Marc-Olaf Jaschke http://git.postgresql.org/pg/commitdiff/4e344c2cf4ff00ca38ea0035bc137dab95fdd0c0
  • Ensure that num_sync is greater than zero in synchronous_standby_names. Previously num_sync could be set to zero and this setting caused an assertion failure. This means that multiple synchronous standbys code should assume that num_sync is greater than zero. Also setting num_sync to zero is nonsense because it's basically the configuration for synchronous replication. If users want not to make transaction commits wait for any standbys, synchronous_standby_names should be emptied to disable synchronous replication instead of setting num_sync to zero. This patch forbids users from setting num_sync to zero in synchronous_standby_names. If zero is specified, an error will happen during processing the parameter settings. Back-patch to 9.6 where multiple synchronous standbys feature was added. Patch by me. Reviewed by Tom Lane. Discussion: <CAHGQGwHWB3izc6cXuFLh5kOcAbFXaRhhgwd-X5PeN9TEjxqXwg@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/93eb619cd35b8adcfe6c86e34ea45d2e8edd322b

Noah Misch pushed:

Correctifs en attente

Pavel Stěhule sent in two more revisions of a patch to implement \g[b]store commands for psql.

Rafia Sabih sent in a patch to implement parallel index-only scans.

Michaël Paquier sent in a patch to fix an issue with strong random on Win32.

Michaël Paquier, Amit Kapila and Ashutosh Sharma traded patches to fix an issue that manifested as a hang in pldebugger.

Amit Kapila sent in another revision of a patch to fix some hash index issues.

Tomas Vondra sent in another revision of a patch to implement multivariate statistics.

Nikita Glukhov sent in a patch to implement recursive processing of nested objects and arrays in json[b]_populate_record[set](), json[b]_to_record[set]().

Tomas Vondra sent in two more revisions of a patch to implement two slab-like memory allocators.

Kyotaro HORIGUCHI sent in another revision of a patch to implement character conversion with radix trees.

Dilip Kumar sent in another revision of a patch to implement parallel bitmap heap scan.

Tom Lane sent in a patch to fix matching of boolean index columns to sort ordering.

Aleksander Alekseev sent in a patch to fix documentation of the timestamp type.

Gilles Darold sent in another revision of a patch to implement pg_current_logfile().

Andrew Borodin sent in a patch to add a sleep sort test for pg_background.

Masahiko Sawada and Fujii Masao traded patches to add quorum commit.

Dilip Kumar sent in another revision of a patch to implement parallel merge join.

Robert Haas sent in a patch to refactor tstate serialization in preparation for making CURRENT_* parallel safe.

Jesper Pedersen sent in a patch to create a pg_catversion() builtin function.

Michaël Paquier, Fujii Masao, and Kyotaro HORIGUCHI traded patches to fix a pg_basebackup crash.

Vladimir Rusinov sent in a patch to rename pg_switch_xlog to pg_switch_wal.

Robert Haas sent in a patch to remove _hash_wrtbuf() and cause those functions which previously called it to do MarkBufferDirty() directly.

Peter Eisentraut, Petr Jelínek, and Steve Singer traded patches to implement logical replication.

Rahila Syed sent in another revision of a patch to turn the types columns with unknown type in views to text.

Dagfinn Ilmari Mannsåker sent in a patch to add GUCs for predicate lock promotion thresholds.

Magnus Hagander sent in a patch to remove the -X option from pg_basebackup.

Magnus Hagander sent in a patch to add missing newlines to error messages in pg_basebackup.

Wesley Massuda sent in a patch to implement composite type NOT NULL constraints.

Alexander Law sent in a patch to speed up documentation builds.

Pavel Stěhule sent in two more revisions of a patch to implement server-side session variables.

Dmitry Ivanov sent in a patch to fix an infelicity between the ancient sql_inheritance GUC and the new native partitions.

Heikki Linnakangas sent in a patch to shorten the window between writing the timeline history file and writing the end-of-recovery record.

Magnus Hagander sent in two revisions of a patch to add a --no-slot option to pg_basebackup.

Heikki Linnakangas sent in a patch to use the "plain:" prefix for plaintext passwords stored in pg_authid.

Amit Kapila sent in two revisions of a patch to fix dirty marking and lock chaining for hash indexes.

Mithun Cy sent in another revision of a patch to cache meta page for hash indexes.

Antonin Houska sent in a patch to simplify some code in basebackup.

Peter Moser sent in another revision of a patch to implement temporal query processing with range types.

Fujii Masao sent in a patch to fix a bug in synchronous replication where there were an invalid number of sync standbys in synchronous_standby_names.

Stephen Frost sent in a patch to fix some infelicities between pg_dump and TRANSFORMs, in passing fixing pg_dump for some ancient versions of PostgreSQL.

Robert Haas sent in a patch to remove the sql_inheritance GUC.

Michaël Paquier sent in a patch to fix some typos in waldender.c and slot.c.

Dmitry Dolgov sent in a patch to implement jsonb_delete with arrays.

Dean Rasheed sent in a patch to fix a CREATE OR REPLACE VIEW bug.

Stas Kelvich sent in a patch to speed up two-phase transactions via a twophase recovery list.

Kevin Grittner sent in another revision of a patch to add transition tables for, among other things, materialized view maintenance and row access in per-statement triggers.

Pavel Stěhule sent in another revision of a patch to implement xmltable().

Peter Eisentraut sent in a patch to add a function to import operation system collations.

par N Bougain le mardi 20 décembre 2016 à 01h10

lundi 12 décembre 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 11 décembre 2016

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

  • Le PGDay.IT 2016 aura lieu à Prato le 13 décembre 2016 : http://pgday.it
  • La PGConf India 2017 aura lieu les 2 & 3 mars 2017 à Bengalore (État du Karnataka en Inde). Les propositions sont attendues sur <papers AT pgconf DOT in> jusqu'au 31 décembre 2016 : http://pgconf.in/
  • PostgreSQL@SCaLE aura lieu les 2 & 3 mars 2017 au centre des conventions de Pasadena, comme partie du SCaLE 15X : http://www.socallinuxexpo.org/scale/15x/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161211224221.GB8506@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Fujii Masao pushed:

  • Fix incorrect output from gin_desc(). Previously gin_desc() displayed incorrect output "unknown action 0" for XLOG_GIN_INSERT and XLOG_GIN_VACUUM_DATA_LEAF_PAGE records with valid actions. The cause of this problem was that gin_desc() wrongly used XLogRecGetData() to extract data from those records. Since they were registered by XLogRegisterBufData(), gin_desc() should have used XLogRecGetBlockData(), instead, like gin_redo(). Also there were other differences about how to treat XLOG_GIN_INSERT record between gin_desc() and gin_redo(). This commit fixes gin_desc() routine so that it treats those records in the same way as gin_redo(). Batch-patch to 9.5 where WAL record format was revamped and XLogRegisterBufData() was added. Reported-By: Andres Freund Reviewed-By: Tom Lane Discussion: <20160509194645.7lewnpw647zegx2m@alap3.anarazel.de> http://git.postgresql.org/pg/commitdiff/5dc851afde8d9ef9947f21799f7a1b08bf0bf812
  • Fix typo in docs. Reported-by: Darko Prelec http://git.postgresql.org/pg/commitdiff/daac8e30eb7874722f277ae3461abe46a39e56ed
  • Improve documentation about pg_stat_replication view. Add the descriptions of possible values in "state" and "sync_state" columns of pg_stat_replication view. Author: Michael Paquier, slightly modified by me Discussion: <CAB7nPqT7APWrvPFZrcjKEHoq4=g3z2ErxtTdojSf+sDALzuemA@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/dfe530a09226a9de80f2b4c3d5f667bf51481c49

Heikki Linnakangas pushed:

  • Fix creation of stand-alone INSTALL.html file. I missed the notice at the top of the file, that plain xref must not be used in installation.sgml. Per buildfarm member guaibasaurus. http://git.postgresql.org/pg/commitdiff/7dd8eb39bd2b9e06eeef038f80ae327efb4a7d55
  • Fix typo in new message in configure. Remove spurious "of", and reformat to fit on a 80 chars wide line. http://git.postgresql.org/pg/commitdiff/44a977f55f33834a2fe0d1d9bd5eeb29ce67e914
  • Fix whitespace. Thomas Munro http://git.postgresql.org/pg/commitdiff/9790b87f594565c11599b2004466169b8c2fd4af
  • Fix query cancellation. In commit fe0a0b59, the datatype used for MyCancelKey and other variables that store cancel keys were changed from long to uint32, but I missed this one. That broke query cancellation on platforms where long is wider than 32 bits. Report by Andres Freund, fix by Michael Paquier. http://git.postgresql.org/pg/commitdiff/81f2e514a9b2d813bb5fd6b62523757aa7a6517f
  • Replace PostmasterRandom() with a stronger source, second attempt. This adds a new routine, pg_strong_random() for generating random bytes, for use in both frontend and backend. At the moment, it's only used in the backend, but the upcoming SCRAM authentication patches need strong random numbers in libpq as well. pg_strong_random() is based on, and replaces, the existing implementation in pgcrypto. It can acquire strong random numbers from a number of sources, depending on what's available: OpenSSL RAND_bytes(), if built with OpenSSL, on Windows, the native cryptographic functions are used, and finally /dev/urandom. Unlike the current pgcrypto function, the source is chosen by configure. That makes it easier to test different implementations, and ensures that we don't accidentally fall back to a less secure implementation, if the primary source fails. All of those methods are quite reliable, it would be pretty surprising for them to fail, so we'd rather find out by failing hard. If no strong random source is available, we fall back to using erand48(), seeded from current timestamp, like PostmasterRandom() was. That isn't cryptographically secure, but allows us to still work on platforms that don't have any of the above stronger sources. Because it's not very secure, the built-in implementation is only used if explicitly requested with --disable-strong-random. This replaces the more complicated Fortuna algorithm we used to have in pgcrypto, which is unfortunate, but all modern platforms have /dev/urandom, so it doesn't seem worth the maintenance effort to keep that. pgcrypto functions that require strong random numbers will be disabled with --disable-strong-random. Original patch by Magnus Hagander, tons of further work by Michael Paquier and me. Discussion: https://www.postgresql.org/message-id/CAB7nPqRy3krN8quR9XujMVVHYtXJ0_60nqgVc6oUk8ygyVkZsA@mail.gmail.com Discussion: https://www.postgresql.org/message-id/CAB7nPqRWkNYRRPJA7-cF+LfroYV10pvjdz6GNvxk-Eee9FypKA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/fe0a0b5993dfe24e4b3bcf52fa64ff41a444b8f1
  • Clean up password authentication code a bit. Commit fe0a0b59, which moved code to do MD5 authentication to a separate CheckMD5Auth() function, left behind a comment that really belongs inside the function, too. Also move the check for db_user_namespace inside the function, seems clearer that way. Now that the md5 salt is passed as argument to md5_crypt_verify, it's a bit silly that it peeks into the Port struct to see if MD5 authentication was used. Seems more straightforward to treat it as an MD5 authentication, if the md5 salt argument is given. And after that, md5_crypt_verify only used the Port argument to look at port->user_name, but that is redundant, because it is also passed as a separate 'role' argument. So remove the Port argument altogether. http://git.postgresql.org/pg/commitdiff/fe7bdf0bf67d8ac360d67fa9740074a2c70e88a4
  • Fix quoting and a compiler warning in dumping partitions. Partition name needs to be quoted in the ATTACH PARTITION command constructed in binary-upgrade mode. Silence compiler warning about set but unused variable, without --enable-cassert. http://git.postgresql.org/pg/commitdiff/2560d244b4c9fc08f1d076b3c40913bec5f7e31f
  • Fix thinko in safeguard for negative availMem. Also, use pass read_buffer_size * numInputTapes rather than just availMem to USEMEM, to be neat. Peter Geoghegan. http://git.postgresql.org/pg/commitdiff/64bc26f90d342ca343f5ba383a97691a58991204
  • Fix accounting of memory needed for merge heap. We allegedly allocated all remaining memory for the read buffers of the sort tapes, but we allocated the merge heap only after that. That means that the allocation of the merge heap was guaranteed to go over the memory limit. Fix by allocating the merge heap first. This makes little difference in practice, because the merge heap is tiny, but let's tidy. While we're at it, add a safeguard for the case that we are already over the limit when allocating the read buffers. That shouldn't happen, but better safe than sorry. The memory accounting error was reported off-list by Peter Geoghegan. http://git.postgresql.org/pg/commitdiff/f7d54f4f7ddf72bf4db1783890b058e758b4b894

Tom Lane pushed:

  • Remove extraneous semicolon from uses of relptr_declare(). If we're going to write a semicolon after calls of relptr_declare(), then we don't need one inside the macro, and removing it suppresses "empty declaration" warnings from pickier compilers (eg pademelon). While at it, we might as well use relptr() inside relptr_declare(), because otherwise that macro would likely go unused altogether. Also improve the comment, which I for one found unclear, and provide a specific example of intended usage. http://git.postgresql.org/pg/commitdiff/3ebf2b45454a5fb74e6516ab4915f7a3d44d544f
  • Put AC_MSG_RESULT() call in the right place. Thinko in ecb0d20a9 --- this needs to go one level further out in the "if" nest. As it stood, nothing got printed in the case of selecting named POSIX semaphores. Cosmetic issue only, but a bug. http://git.postgresql.org/pg/commitdiff/c648f058319a59ad591dd9d1b0c48dfd655d063a
  • Fix unsafe assumption that struct timeval.tv_sec is a "long". It typically is a "long", but it seems possible that on some platforms it wouldn't be. In any case, this silences a compiler warning on OpenBSD (cf buildfarm member curculio). While at it, use snprintf not sprintf. This format string couldn't possibly overrun the supplied buffer, but that doesn't seem like a good reason not to use the safer style. Oversight in commit f828654e1. Back-patch to 9.6 where that came in. http://git.postgresql.org/pg/commitdiff/0645dacc371da6169b06934e3bd238c5f770fe25
  • Handle empty or all-blank PAGER setting more sanely in psql. If the PAGER environment variable is set but contains an empty string, psql would pass it to "sh" which would silently exit, causing whatever query output we were printing to vanish entirely. This is quite mystifying; it took a long time for us to figure out that this was the cause of Joseph Brenner's trouble report. Rather than allowing that to happen, we should treat this as another way to specify "no pager". (We could alternatively treat it as selecting the default pager, but it seems more likely that the former is what the user meant to achieve by setting PAGER this way.) Nonempty, but all-white-space, PAGER values have the same behavior, and it's pretty easy to test for that, so let's handle that case the same way. Most other cases of faulty PAGER values will result in the shell printing some kind of complaint to stderr, which should be enough to diagnose the problem, so we don't need to work harder than this. (Note that there's been an intentional decision not to be very chatty about apparent failure returns from the pager process, since that may happen if, eg, the user quits the pager with control-C or some such. I'd just as soon not start splitting hairs about which exit codes might merit making our own report.) libpq's old PQprint() function was already on board with ignoring empty PAGER values, but for consistency, make it ignore all-white-space values as well. It's been like this a long time, so back-patch to all supported branches. Discussion: https://postgr.es/m/CAFfgvXWLOE2novHzYjmQK8-J6TmHz42G8f3X0SORM44+stUGmw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/18f8f784cbbf96ef77eb8943b466b26605824c14
  • Restore psql's SIGPIPE setting if popen() fails. Ancient oversight in PageOutput(): if popen() fails, we'd better reset the SIGPIPE handler before returning stdout, because ClosePager() won't. Noticed while fixing the empty-PAGER issue. http://git.postgresql.org/pg/commitdiff/b7e1ae2328f7d5a88d8916d78b4561d8ef16f99b
  • Fix reporting of column typmods for multi-row VALUES constructs. expandRTE() and get_rte_attribute_type() reported the exprType() and exprTypmod() values of the expressions in the first row of the VALUES as being the column type/typmod returned by the VALUES RTE. That's fine for the data type, since we coerce all expressions in a column to have the same common type. But we don't coerce them to have a common typmod, so it was possible for rows after the first one to return values that violate the claimed column typmod. This leads to the incorrect result seen in bug #14448 from Hassan Mahmood, as well as some other corner-case misbehaviors. The desired behavior is the same as we use in other type-unification cases: report the common typmod if there is one, but otherwise return -1 indicating no particular constraint. It's cheap for transformValuesClause to determine the common typmod while transforming a multi-row VALUES, but it'd be less cheap for expandRTE() and get_rte_attribute_type() to re-determine that info every time they're asked --- possibly a lot less cheap, if the VALUES has many rows. Therefore, the best fix is to record the common typmods explicitly in a list in the VALUES RTE, as we were already doing for column collations. This looks quite a bit like what we're doing for CTE RTEs, so we can save a little bit of space and code by unifying the representation for those two RTE types. They both now share coltypes/coltypmods/colcollations fields. (At some point it might seem desirable to populate those fields for all RTE types; but right now it looks like constructing them for other RTE types would add more code and cycles than it would save.) The RTE change requires a catversion bump, so this fix is only usable in HEAD. If we fix this at all in the back branches, the patch will need to look quite different. Report: https://postgr.es/m/20161205143037.4377.60754@wrigleys.postgresql.org Discussion: https://postgr.es/m/27429.1480968538@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/0b78106cd4651ad7867036a463ec743f6d3d2e86
  • Be more careful about Python refcounts while creating exception objects. PLy_generate_spi_exceptions neglected to do Py_INCREF on the new exception objects, evidently supposing that PyModule_AddObject would do that --- but it doesn't. This left us in a situation where a Python garbage collection cycle could result in deletion of exception object(s), causing server crashes or wrong answers if the exception objects are used later in the session. In addition, PLy_generate_spi_exceptions didn't bother to test for a null result from PyErr_NewException, which at best is inconsistent with the code in PLy_add_exceptions. And PLy_add_exceptions, while it did do Py_INCREF on the exceptions it makes, waited to do that till after some PyModule_AddObject calls, creating a similar risk for failure if garbage collection happened within those calls. To fix, refactor to have just one piece of code that creates an exception object and adds it to the spiexceptions module, bumping the refcount first. Also, let's add an additional refcount to represent the pointer we're going to store in a C global variable or hash table. This should only matter if the user does something weird like delete the spiexceptions Python module, but lack of paranoia has caused us enough problems in PL/Python already. The fact that PyModule_AddObject doesn't do a Py_INCREF of its own explains the need for the Py_INCREF added in commit 4c966d920, so we can improve the comment about that; also, this means we really want to do that before not after the PyModule_AddObject call. The missing Py_INCREF in PLy_generate_spi_exceptions was reported and diagnosed by Rafa de la Torre; the other fixes by me. Back-patch to all supported branches. Discussion: https://postgr.es/m/CA+Fz15kR1OXZv43mDrJb3XY+1MuQYWhx5kx3ea6BRKQp6ezGkg@mail.gmail.com http://git.postgresql.org/pg/commitdiff/9cda81f0056ca488dbd6cded64db1238aed816b2
  • Prevent crash when ts_rewrite() replaces a non-top-level subtree with null. When ts_rewrite()'s replacement argument is an empty tsquery, it's supposed to simplify any operator nodes whose operand(s) become NULL; but it failed to do that reliably, because dropvoidsubtree() only examined the top level of the result tree. Rather than make a second recursive pass, let's just give the responsibility to dofindsubquery() to simplify while it's doing the main replacement pass. Per report from Andreas Seltenreich. Artur Zakirov, with some cosmetic changes by me. Back-patch to all supported branches. Discussion: https://postgr.es/m/8737i01dew.fsf@credativ.de http://git.postgresql.org/pg/commitdiff/0eaaaf00e296c2048b868b7c1d3c12c0eae6dd12

Stephen Frost pushed:

Ãlvaro Herrera pushed:

  • Fix crasher bug in array_position(s) array_position and its cousin array_positions were caching the element type equality function's FmgrInfo without being careful enough to put it in a long-lived context. This is obviously broken but it didn't matter in most cases; only when using arrays of records (involving record_eq) it becomes a problem. The fix is to ensure that the type's equality function's FmgrInfo is cached in the array_position's flinfo->fn_mcxt rather than the current memory context. Apart from record types, the only other case that seems complex enough to possibly cause the same problem are range types. I didn't find a way to reproduce the problem with those, so I only include the test case submitted with the bug report as regression test. Bug report and patch: Junseok Yang Discussion: https://postgr.es/m/CAE+byMupUURYiZ6bKYgMZb9pgV1CYAijJGqWj-90W=nS7uEOeA@mail.gmail.com Backpatch to 9.5, where array_position appeared. http://git.postgresql.org/pg/commitdiff/a73491e5fee88f5db70d69e81fa45060b6ed3682

Correctifs en attente

Michaël Paquier sent in another revision of a patch to enable SIGHUP to reload SSL certificates.

Amit Kapila sent in another revision of a patch to enable write-ahead logging (WAL) for hash indexes.

KaiGai Kohei sent in another revision of a patch to pass down limit bounds for ForeignScan and CustomScan.

Pavel StÄ›hule and Ãlvaro Herrera traded patches to add xmltable().

Michaël Paquier sent in three more revisions of a patch to fix a pgcrypto compilation error caused by stack-allocated EVP_CIPHER_CTX.

Kyotaro HORIGUCHI sent in a patch to change the type for map files not to be void * any more, and in passing clean up convutils.pm.

Masahiko Sawada sent in a patch to reduce the maximum autovacuum_vacuum and analyze_scale_factor from 100 to 1.

Mithun Cy sent in a patch to fix an issue with SSL vs. multi-host URLs in libpq.

Mithun Cy sent in another revision of a patch to cache hash index meta pages.

Andres Freund sent in a WIP patch to use, among other techniques, JIT compilation, to speed up expression processing and tuple deforming.

Craig Ringer sent in another revision of a patch to enable logical decoding on standby.

Rahila Syed sent in a patch to assign valid collations for SET operations on queries with UNKNOWN types.

Heikki Linnakangas sent in a flock of patches to create the infrastructure for, and implement, SCRAM auth.

Gilles Darold and Karl O. Pinc traded patches to implement pg_current_logfile().

Peter Moser sent in another revision of a patch to implement temporal query processing with range types.

Stephen Frost sent in a patch to fix some infelicities between pg_dump and both TRANSFORMs and CASTs.

Michaël Paquier sent in another revision of a patch to fix how unlogged tables are cleaned up.

Etsuro Fujita sent in another revision of a patch to push more FULL JOINs down when using foreign data wrappers.

Aleksander Alekseev sent in two revisions of a patch to clarify some code by using pg_str_containsonly.

Ashutosh Sharma sent in a patch to fix a bug that cause pldebugger to hang.

Peter Eisentraut and Petr Jelínek traded patches to support logical replication in core.

Masahiko Sawada sent in another revision of a patch to implement quorum commit for multiple synchronous replication.

Amit Langote sent in a patch to document the behavior of UPDATE and DELETE under built-in partitioning.

Fabrízio de Royes Mello sent in a patch to fix a breakage of psql's \d when connecting to older versions of PostgreSQL.

Peter Geoghegan sent in a patch to remove should_free tuplesort routine arguments, avoid copying within tuplesort_gettupleslot(), and add valgrind suppression for logtape_write.

Dilip Kumar sent in a patch to implement parallel merge join.

Amit Kapila sent in a patch to fix some issues in hash indexes.

Tom Lane sent in a patch to back-patch use of unnamed POSIX semaphores for Linux.

Petr Jelínek sent in a patch to skip unnecessary snapshot builds.

Pavel Stěhule sent in a patch to add \gstore \gstore_binary to psql.

Mateusz Stefek sent in a patch to optimize index-only scans with filter conditions.

par N Bougain le lundi 12 décembre 2016 à 23h44

samedi 10 décembre 2016

Guillaume Lelarge

Version 1.3 de mon livre : PostgreSQL - Architecture et notions avancées

Mon livre, PostgreSQL - Architecture et notions avancées, a été écrit pendant le développement de la version 9.5. Il est sorti en version finale un peu après la sortie de la version 9.5, donc en plein développement de la 9.6. À la sortie de la version beta de la 9.6, je me suis mis à travailler sur une mise à jour du livre, incluant des corrections suite à des commentaires qui m'étaient parvenus, ainsi que de nombreux ajouts pour traiter les nouveautés de la version 9.6. Le résultat final est disponible depuis hier. Comme il ne s'agit pas d'une nouvelle édition complète, il est disponible gratuitement en téléchargement aux personnes qui ont acheté une version électronique précédente. Ceux qui commanderaient maintenant la version papier aurait cette nouvelle version, intitulée 1.3.

Évidemment, nous allons continuer la mise à jour du livre pour qu'il ne perde pas en intérêt. Une (vraie) deuxième édition sera disponible en fin d'année prochaine, après la sortie de la version 10. Cette version promet de nombreuses nouveautés très intéressantes, comme tout dernièrement un partitionnement mieux intégré dans le cœur de PostgreSQL.

par Guillaume Lelarge le samedi 10 décembre 2016 à 15h36

mercredi 7 décembre 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 4 décembre 2016

[ndt: MeetUp en préparation pour le 20 décembre à Lyon : http://www.meetup.com/fr-FR/PostgreSQL-Lyon-User-Group/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en décembre

PostgreSQL Local

  • CHAR(16) aura lieu à New York le 6 décembre 2016 : http://charconference.org/
  • Le PGDay.IT 2016 aura lieu à Prato le 13 décembre 2016 : http://pgday.it
  • La PGConf India 2017 aura lieu les 2 & 3 mars 2017 à Bengalore (État du Karnataka en Inde). Les propositions sont attendues sur <papers AT pgconf DOT in> jusqu'au 31 décembre 2016 : http://pgconf.in/
  • PostgreSQL@SCaLE aura lieu les 2 & 3 mars 2017 au centre des conventions de Pasadena, comme partie du SCaLE 15X : http://www.socallinuxexpo.org/scale/15x/

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161205004347.GA329@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Code review for early drop of orphaned temp relations in autovacuum. Commit a734fd5d1 exposed some race conditions that existed previously in the autovac code, but were basically harmless because autovac would not try to delete orphaned relations immediately. Specifically, the test for orphaned-ness was made on a pg_class tuple that might be dead by now, allowing autovac to try to remove a table that the owning backend had just finished deleting. This resulted in a hard crash due to inadequate caution about accessing the table's catalog entries without any lock. We must take a relation lock and then recheck whether the table is still present and still looks deletable before we do anything. Also, it seemed to me that deleting multiple tables per transaction, and trying to continue after errors, represented unjustifiable complexity. We do not expect this code path to be taken often in the field, nor even during testing, which means that prioritizing performance over correctness is a bad tradeoff. Rip all that out in favor of just starting a new transaction after each successful temp table deletion. If we're unlucky enough to get an error, which shouldn't happen anyway now that we're being more cautious, let the autovacuum worker fail as it normally would. In passing, improve the order of operations in the initial scan loop. Now that we don't care about whether a temp table is a wraparound hazard, there's no need to perform extract_autovac_opts, get_pgstat_tabentry_relid, or relation_needs_vacanalyze for temp tables. Also, if GetTempNamespaceBackendId returns InvalidBackendId (indicating it doesn't recognize the schema as temp), treat that as meaning it's NOT an orphaned temp table, not that it IS one, which is what happened before because BackendIdGetProc necessarily failed. The case really shouldn't come up for a table that has RELPERSISTENCE_TEMP, but the consequences if it did seem undesirable. (This might represent a back-patchable bug fix; not sure if it's worth the trouble.) Discussion: https://postgr.es/m/21299.1480272347@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/dafa0848da11260e9510c699e7060171336cb550
  • Fix busted tab-completion pattern for ALTER TABLE t ALTER c DROP ... Evidently a thinko in commit 9b181b036. Kyotaro Horiguchi http://git.postgresql.org/pg/commitdiff/404e667580fec0ab9a89a38eaa8643ee7227fdb8
  • Fix estimate_expression_value to constant-fold SQLValueFunction nodes. Oversight in my commit 0bb51aa96. Noted while poking at a recent bug report --- HEAD's estimates for a query using CURRENT_DATE were unexpectedly much worse than 9.6's. http://git.postgresql.org/pg/commitdiff/4e20511d5b8ba427730e09be45f9458f667f9c1e
  • Fix incorrect variable type in set_rel_consider_parallel(). func_parallel() returns char not Oid. Harmless, but still wrong. Amit Langote http://git.postgresql.org/pg/commitdiff/d6c8b34e956864d52780f0db9f8cfe3b2f4411b0
  • Add uuid to the set of types supported by contrib/btree_gist. Paul Jungwirth, reviewed and hacked on by Teodor Sigaev, Ildus Kurbangaliev, Adam Brusselback, Chris Bandy, and myself. Discussion: https://postgr.es/m/CA+renyUEE29=X01JXdz8_TQvo6n9=2XoEBBRnQ8rkLyr+kjPxQ@mail.gmail.com Discussion: https://postgr.es/m/55F6EE82.8080209@sigaev.ru http://git.postgresql.org/pg/commitdiff/11da83a0e70d32ed0e06a5c948cd8343f8ad5102
  • Test all contrib-created operator classes with amvalidate. I'd supposed that people would do this manually when creating new operator classes, but the folly of that was exposed today. The tests seem fast enough that we can just apply them during the normal regression tests. contrib/isn fails the checks for lack of complete sets of cross-type operators. That's a nice-to-have policy rather than a functional requirement, so leave it as-is, but insert ORDER BY in the query to ensure consistent cross-platform output. Discussion: https://postgr.es/m/7076.1480446837@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/ade49c605f1d8f517497829363f8b83f69c73a60
  • Improve eqjoinsel_semi's behavior for small inner relations with no stats. If we don't have any MCV statistics for the inner relation, and we don't trust its numdistinct estimate either, eqjoinsel_semi falls back to a very conservative estimate (that 50% of the outer rows have matches). This is particularly problematic if the inner relation is completely empty, since then even an explicit ANALYZE won't produce any pg_statistic entries, so there's no way to budge the planner off the bad estimate. We'd produce a better estimate in such cases if we used the nd2/nd1 selectivity heuristic, so an easy fix is to treat the nd2 estimate as non-default if we derive it from clamping to the inner rel's rowcount estimate. This won't fix every related case (mainly because the rowcount estimate might be larger than DEFAULT_NUM_DISTINCT), but it seems like a sane extension of the existing logic, so let's apply the change in HEAD and see if anyone complains. Per bug #14438 from Nikolay Nikitin. Report: https://postgr.es/m/20161128182113.6527.58926@wrigleys.postgresql.org Discussion: https://postgr.es/m/31089.1480384713@sss.pgh.pa.us http://git.postgresql.org/pg/commitdiff/ca5f88502096a39f831da74eb02ec9eb3310616d
  • Fix bogus handling of JOIN_UNIQUE_OUTER/INNER cases for parallel joins. consider_parallel_nestloop passed the wrong jointype down to its subroutines for JOIN_UNIQUE_INNER cases (it should pass JOIN_INNER), and it thought that it could pass paths other than innerrel->cheapest_total_path to create_unique_path, which create_unique_path is not on board with. These bugs would lead to assertion failures or other errors, suggesting that this code path hasn't been tested much. hash_inner_and_outer's code for parallel join effectively treated both JOIN_UNIQUE_OUTER and JOIN_UNIQUE_INNER the same as JOIN_INNER (for different reasons :-(), leading to incorrect plans that treated a semijoin as if it were a plain join. Michael Day submitted a test case demonstrating that hash_inner_and_outer failed for JOIN_UNIQUE_OUTER, and I found the other cases through code review. Report: https://postgr.es/m/D0E8A029-D1AC-42E8-979A-5DE4A77E4413@rcmail.com http://git.postgresql.org/pg/commitdiff/41e2b84ce1b89ae0125dd34318c56aa51386e2a2
  • Doc: improve description of trim() and related functions. Per bug #14441 from Mark Pether, the documentation could be misread, mainly because some of the examples failed to show what happens with a multicharacter "characters to trim" string. Also, while the text description in most of these entries was fairly clear that the "characters" argument is a set of characters not a substring to match, some of them used variant wording that was a bit less clear. trim() itself suffered from both deficiencies and was thus pretty misinterpretable. Also fix failure to explain which of LEADING/TRAILING/BOTH is the default. Discussion: https://postgr.es/m/20161130011710.6539.53657@wrigleys.postgresql.org http://git.postgresql.org/pg/commitdiff/213c0f2d7880f78c710127920cf4bf7017e0fa57
  • Delete deleteWhatDependsOn() in favor of more performDeletion() flag bits. deleteWhatDependsOn() had grown an uncomfortably large number of assumptions about what it's used for. There are actually only two minor differences between what it does and what a regular performDeletion() call can do, so let's invent additional bits in performDeletion's existing flags argument that specify those behaviors, and get rid of deleteWhatDependsOn() as such. (We'd probably have done it this way from the start, except that performDeletion didn't originally have a flags argument, IIRC.) Also, add a SKIP_EXTENSIONS flag bit that prevents ever recursing to an extension, and use that when dropping temporary objects at session end. This provides a more general solution to the problem addressed in a hacky way in commit 08dd23cec: if an extension script creates temp objects and forgets to remove them again, the whole extension went away when its contained temp objects were deleted. The previous solution only covered temp relations, but this solves it for all object types. These changes require minor additions in dependency.c to pass the flags to subroutines that previously didn't get them, but it's still a net savings of code, and it seems cleaner than before. Having done this, revert the special-case code added in 08dd23cec that prevented addition of pg_depend records for temp table extension membership, because that caused its own oddities: dropping an extension that had created such a table didn't automatically remove the table, leading to a failure if the table had another dependency on the extension (such as use of an extension data type), or to a duplicate-name failure if you then tried to recreate the extension. But we keep the part that prevents the pg_temp_nnn schema from becoming an extension member; we never want that to happen. Add a regression test case covering these behaviors. Although this fixes some arguable bugs, we've heard few field complaints, and any such problems are easily worked around by explicitly dropping temp objects at the end of extension scripts (which seems like good practice anyway). So I won't risk a back-patch. Discussion: https://postgr.es/m/e51f4311-f483-4dd0-1ccc-abec3c405110@BlueTreble.com http://git.postgresql.org/pg/commitdiff/b3427dade14cc31eb48740bc9ea98b5954470b24
  • Fix broken wait-for-previous-process-to-exit loop in regression test. Must do pg_stat_clear_snapshot() inside test's loop, or our snapshot of pg_stat_activity will never change :-(. Thinko in b3427dade -- evidently my workstation never really iterated the loop in testing. Per buildfarm. http://git.postgresql.org/pg/commitdiff/19fcc0058ecc8e5eb756547006bc1b24a93cbb80
  • Don't mess up pstate->p_next_resno in transformOnConflictClause(). transformOnConflictClause incremented p_next_resno while generating the phony targetlist for the EXCLUDED pseudo-rel. Then that field got incremented some more during transformTargetList, possibly leading to free_parsestate concluding that we'd overrun the allowed length of a tlist, as reported by Justin Pryzby. We could fix this by resetting p_next_resno to 1 after using it for the EXCLUDED pseudo-rel tlist, but it seems easier and less coupled to other places if we just don't use that field at all in this loop. (Note that this doesn't change anything about the resnos that end up appearing in the main target list, because those are all replaced with target-column numbers by updateTargetListEntry.) In passing, fix incorrect type OID assigned to the whole-row Var for "EXCLUDED.*" (somehow this escaped having any bad consequences so far, but it's certainly wrong); remove useless assignment to var->location; pstrdup the column names in case of a relcache flush; and improve nearby comments. Back-patch to 9.5 where ON CONFLICT was introduced. Report: https://postgr.es/m/20161204163237.GA8030@telsasoft.com http://git.postgresql.org/pg/commitdiff/3850723208888a6fe90b75ebf692af79119f4158

Ãlvaro Herrera pushed:

  • Fix get_relation_info name typo'ed in a comment. Plus add a missing comment about this in get_relation_info itself. Author: Amit Langote. Discussion: https://postgr.es/m/e46c0569-0449-afa0-e2fe-f3776e4b3fd5@lab.ntt.co.jp http://git.postgresql.org/pg/commitdiff/eb6814168862ef0ff44716467aacdebcef87415a
  • Permit dump/reload of not-too-large >1GB tuples. Our documentation states that our maximum field size is 1 GB, and that our maximum row size of 1.6 TB. However, while this might be attainable in theory with enough contortions, it is not workable in practice; for starters, pg_dump fails to dump tables containing rows larger than 1 GB, even if individual columns are well below the limit; and even if one does manage to manufacture a dump file containing a row that large, the server refuses to load it anyway. This commit enables dumping and reloading of such tuples, provided two conditions are met: 1. no single column is larger than 1 GB (in output size -- for bytea this includes the formatting overhead) 2. the whole row is not larger than 2 GB There are three related changes to enable this: a. StringInfo's API now has two additional functions that allow creating a string that grows beyond the typical 1GB limit (and "long" string). ABI compatibility is maintained. We still limit these strings to 2 GB, though, for reasons explained below. b. COPY now uses long StringInfos, so that pg_dump doesn't choke trying to emit rows longer than 1GB. c. heap_form_tuple now uses the MCXT_ALLOW_HUGE flag in its allocation for the input tuple, which means that large tuples are accepted on input. Note that at this point we do not apply any further limit to the input tuple size. The main reason to limit to 2 GB is that the FE/BE protocol uses 32 bit length words to describe each row; and because the documentation is ambiguous on its signedness and libpq does consider it signed, we cannot use the highest-order bit. Additionally, the StringInfo API uses "int" (which is 4 bytes wide in most platforms) in many places, so we'd need to change that API too in order to improve, which has lots of fallout. Backpatch to 9.5, which is the oldest that has MemoryContextAllocExtended, a necessary piece of infrastructure. We could apply to 9.4 with very minimal additional effort, but any further than that would require backpatching "huge" allocations too. This is the largest set of changes we could find that can be back-patched without breaking compatibility with existing systems. Fixing a bigger set of problems (for example, dumping tuples bigger than 2GB, or dumping fields bigger than 1GB) would require changing the FE/BE protocol and/or changing the StringInfo API in an ABI-incompatible way, neither of which would be back-patchable. Authors: Daniel Vérité, Ãlvaro Herrera. Reviewed by: Tomas Vondra. Discussion: https://postgr.es/m/20160229183023.GA286012@alvherre.pgsql http://git.postgresql.org/pg/commitdiff/fa2fa995528023b2e6ba1108f2f47558c6b66dcd
  • Fix Windows build for 78c8c814390f. Author: Petr Jelínek http://git.postgresql.org/pg/commitdiff/5714931b075b2dc8994b6e464ea3845c33ecf416
  • Fix outdated comments Commit 597a87ccc9a6f neglected to update some comments; fix. Report and patch by Thomas Munro. Reviewed by Petr Jelínek. http://git.postgresql.org/pg/commitdiff/5e5986b6cbebcb57e6c95463031eef01e099e7e1

Stephen Frost pushed:

Robert Haas pushed:

  • libpq: Add target_session_attrs parameter. Commit 274bb2b3857cc987cfa21d14775cae9b0dababa5 made it possible to specify multiple IPs in a connection string, but that's not good enough for the case where you have a read-write master and a bunch of read-only standbys and want to connect to whichever server is the master at the current time. This commit allows that, by making it possible to specify target_session_attrs=read-write as a connection parameter. There was extensive discussion of the best name for the connection parameter and its values as well as the best way to distinguish master and standbys. For now, adopt the same solution as JDBC: if the user wants a read-write connection, issue 'show transaction_read_only' and rejection the connection if the result is 'on'. In the future, we could add additional values of this new target_session_attrs parameter that issue different queries; or we might have some way of distinguishing the server type without resorting to an SQL query; but right now, we have this, and that's (hopefully) a good start. Victor Wagner and Mithun Cy. Design review by Ãlvaro Herrera, Catalin Iacob, Takayuki Tsunakawa, and Craig Ringer; code review by me. I changed Mithun's patch to skip all remaining IPs for a host if we reject a connection based on this new parameter, rewrote the documentation, and did some other cosmetic cleanup. Discussion: http://postgr.es/m/CAD__OuhqPRGpcsfwPHz_PDqAGkoqS1UvnUnOnAB-LBWBW=wu4A@mail.gmail.com http://git.postgresql.org/pg/commitdiff/721f7bd3cbccaf8c07cad2707826b83f84694832
  • Improve hash index bucket split behavior. Previously, the right to split a bucket was represented by a heavyweight lock on the page number of the primary bucket page. Unfortunately, this meant that every scan needed to take a heavyweight lock on that bucket also, which was bad for concurrency. Instead, use a cleanup lock on the primary bucket page to indicate the right to begin a split, so that scans only need to retain a pin on that page, which is they would have to acquire anyway, and which is also much cheaper. In addition to reducing the locking cost, this also avoids locking out scans and inserts for the entire lifetime of the split: while the new bucket is being populated with copies of the appropriate tuples from the old bucket, scans and inserts can happen in parallel. There are minor concurrency improvements for vacuum operations as well, though the situation there is still far from ideal. This patch also removes the unworldly assumption that a split will never be interrupted. With the new code, a split is done in a series of small steps and the system can pick up where it left off if it is interrupted prior to completion. While this patch does not itself add write-ahead logging for hash indexes, it is clearly a necessary first step, since one of the things that could interrupt a split is the removal of electrical power from the machine performing it. Amit Kapila. I wrote the original design on which this patch is based, and did a good bit of work on the comments and README through multiple rounds of review, but all of the code is Amit's. Also reviewed by Jesper Pedersen, Jeff Janes, and others. Discussion: http://postgr.es/m/CAA4eK1LfzcZYxLoXS874Ad0+S-ZM60U9bwcyiUZx9mHZ-KCWhw@mail.gmail.com http://git.postgresql.org/pg/commitdiff/6d46f4783efe457f74816a75173eb23ed8930020
  • libpq: Fix inadvertent change in PQhost() behavior. Commit 274bb2b3857cc987cfa21d14775cae9b0dababa5 caused PQhost() to return the value of the hostaddr parameter rather than the relevant host when the latter parameter was specified. That's wrong. Commit 9a1d0af4ad2cbd419115b453d811c141b80d872b then amplified the damage by using PQhost() in more places, so that the SSL test suite started failing. Report by Andreas Karlsson; patch by me. http://git.postgresql.org/pg/commitdiff/11003eb55658df0caf183eef69c7a97d56a4f2d7
  • Add max_parallel_workers GUC. Increase the default value of the existing max_worker_processes GUC from 8 to 16, and add a new max_parallel_workers GUC with a maximum of 8. This way, even if the maximum amount of parallel query is happening, there is still room for background workers that do other things, as originally envisioned when max_worker_processes was added. Julien Rouhaud, reviewed by Amit Kapila and by revised by me. http://git.postgresql.org/pg/commitdiff/b460f5d6693103076dc554aa7cbb96e1e53074f9
  • Clarify that pg_stat_activity.query has a length limit. There was always documentation of the GUC that controlled what the limit actually was, but previously the documentation of the field itself made no mention of that limit. Ian Barwick http://git.postgresql.org/pg/commitdiff/e63d41498837667a4e2e0a4b9416bfda28c722d6
  • Add a crude facility for dealing with relative pointers. C doesn't have any sort of built-in understanding of a pointer relative to some arbitrary base address, but dynamic shared memory segments can be mapped at different addresses in different processes, so any sort of shared data structure stored within a dynamic shared memory segment can't use absolute pointers. We could use something like Size to represent a relative pointer, but then the compiler provides no type-checking. Use stupid macro tricks to get some type-checking. Patch originally by me. Concept suggested by Andres Freund. Recently resubmitted as part of Thomas Munro's work on dynamic shared memory allocation. Discussion: 20131205144434.GG12398@alap2.anarazel.de Discussion: CAEepm=1z5WLuNoJ80PaCvz6EtG9dN0j-KuHcHtU6QEfcPP5-qA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/fbc1c12a94a638cf4f577fef158175e22ab824a3
  • Management of free memory pages. This is intended as infrastructure for a full-fledged allocator for dynamic shared memory. The interface looks a bit like a real allocator, but only supports allocating and freeing memory in multiples of the 4kB page size. Further, to free memory, you must know the size of the span you wish to free, in pages. While these are make it unsuitable as an allocator in and of itself, it still serves as very useful scaffolding for a full-fledged allocator. Robert Haas and Thomas Munro. This code is mostly the same as my 2014 submission, but Thomas fixed quite a few bugs and made some changes to the interface. Discussion: CA+TgmobkeWptGwiNa+SGFWsTLzTzD-CeLz0KcE-y6LFgoUus4A@mail.gmail.com Discussion: CAEepm=1z5WLuNoJ80PaCvz6EtG9dN0j-KuHcHtU6QEfcPP5-qA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/13e14a78ea15f4c581cae408b5010c13961c96de
  • Introduce dynamic shared memory areas. Programmers discovered decades ago that it was useful to have a simple interface for allocating and freeing memory, which is why malloc() and free() were invented. Unfortunately, those handy tools don't work with dynamic shared memory segments because those are specific to PostgreSQL and are not necessarily mapped at the same address in every cooperating process. So invent our own allocator instead. This makes it possible for processes cooperating as part of parallel query execution to allocate and free chunks of memory without having to reserve them prior to the start of execution. It could also be used for longer lived objects; for example, we could consider storing data for pg_stat_statements or the stats collector in shared memory using these interfaces, rather than writing them to files. Basically, anything that needs shared memory but can't predict in advance how much it's going to need might find this useful. Thomas Munro and Robert Haas. The original code (of mine) on which Thomas based his work was actually designed to be a new backend-local memory allocator for PostgreSQL, but that hasn't gone anywhere - or not yet, anyway. Thomas took that work and performed major refactoring and extensive modifications to make it work with dynamic shared memory, including the addition of appropriate locking. Discussion: CA+TgmobkeWptGwiNa+SGFWsTLzTzD-CeLz0KcE-y6LFgoUus4A@mail.gmail.com Discussion: CAEepm=1z5WLuNoJ80PaCvz6EtG9dN0j-KuHcHtU6QEfcPP5-qA@mail.gmail.com http://git.postgresql.org/pg/commitdiff/13df76a537cca3b8884911d8fdf7c89a457a8dd3
  • Fix thinko in b3427dade14cc31eb48740bc9ea98b5954470b24. http://git.postgresql.org/pg/commitdiff/767a9039d79e35c78afa39cf5ffa3b485a2e3a5b

Peter Eisentraut pushed:

Heikki Linnakangas pushed:

Andres Freund pushed:

  • Perform one only projection to compute agg arguments. Previously we did a ExecProject() for each individual aggregate argument. That turned out to be a performance bottleneck in queries with multiple aggregates. Doing all the argument computations in one ExecProject() is quite a bit cheaper because ExecProject's fastpath can do the work at once in a relatively tight loop, and because it can get all the required columns with a single slot_getsomeattr and save some other redundant setup costs. Author: Andres Freund. Reviewed-By: Heikki Linnakangas. Discussion: https://postgr.es/m/20161103110721.h5i5t5saxfk5eeik@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/8ed3f11bb045ad7a3607690be668dbd5b3cc31d7
  • User narrower representative tuples in the hash-agg hashtable. So far the hashtable stored representative tuples in the form of its input slot, with all columns in the hashtable that are not needed (i.e. not grouped upon or functionally dependent) set to NULL. Thats good for saving memory, but it turns out that having tuples full of NULL isn't free. slot_deform_tuple is faster if there's no NULL bitmap even if no NULLs are encountered, and skipping over leading NULLs isn't free. So compute a separate tuple descriptor that only contains the needed columns. As columns have already been moved in/out the slot for the hashtable that does not imply additional per-row overhead. Author: Andres Freund. Reviewed-By: Heikki Linnakangas. Discussion: https://postgr.es/m/20161103110721.h5i5t5saxfk5eeik@alap3.anarazel.de http://git.postgresql.org/pg/commitdiff/fc4b3dea2950e4f6081f1ed2380f82c9efd672e0

Michael Meskes pushed:

Noah Misch pushed:

  • Remove wrong CloseHandle() call. In accordance with its own documentation, invoke CloseHandle() only when directed in the documentation for the function that furnished the handle. GetModuleHandle() does not so direct. We have been issuing this call only in the rare event that a CRT DLL contains no "_putenv" symbol, so lack of bug reports is uninformative. Back-patch to 9.2 (all supported versions). Christian Ullrich, reviewed by Michael Paquier. http://git.postgresql.org/pg/commitdiff/b37da1e8a0e46ae12415fafd3ea441fc3546cf2f
  • Refine win32env.c cosmetics. Replace use of plain 0 as a null pointer constant. In comments, update terminology and lessen redundancy. Back-patch to 9.2 (all supported versions) for the convenience of back-patching the next two commits. Christian Ullrich and Noah Misch, reviewed (in earlier versions) by Michael Paquier. http://git.postgresql.org/pg/commitdiff/a9d9208c15de4933f89e5b6ac1d9ef0efd299162
  • Make pgwin32_putenv() visit debug CRTs. This has no effect in the most conventional case, where no relevant DLL uses a debug build. For an example where it does matter, given a debug build of MIT Kerberos, the krb_server_keyfile parameter usually had no effect. Since nobody wants a Heisenbug, back-patch to 9.2 (all supported versions). Christian Ullrich, reviewed by Michael Paquier. http://git.postgresql.org/pg/commitdiff/95b9b8a3977f854e0bfb2a5614b699b7774ae673
  • Make pgwin32_putenv() follow DLL loading and unloading. Until now, the first putenv() call of a given postgres.exe process would cache the set of loaded CRTs. If a CRT unloaded after that call, the next putenv() would crash. That risk was largely theoretical, because the first putenv() precedes all PostgreSQL-initiated module loading. However, this might explain bad interactions with antivirus and other software that injects threads asynchronously. If an additional CRT loaded after the first putenv(), pgwin32_putenv() would not discover it. That CRT would have all environment changes predating its load, but it would not receive later PostgreSQL-initiated changes. An additional CRT loading concurrently with the first putenv() might miss that change in addition to missing later changes. Fix all those problems. This removes the cache mechanism from pgwin32_putenv(); the cost, less than 100 μs per backend startup, is negligible. No resulting misbehavior was known to be user-visible given the core distribution alone, but one can readily construct an affected extension module. No back-patch given the lack of complaints and the potential for behavior changes in non-PostgreSQL code running in the backend. Christian Ullrich, reviewed by Michael Paquier. http://git.postgresql.org/pg/commitdiff/202dbdbe41e1b1085a4d69c96bca9a52e634b196
  • Make pgwin32_putenv() probe every known CRT, regardless of compiler. This extends to MinGW builds the provision for MSVC-built libraries to see putenv() effects. Doing so repairs, for example, the handling of the krb_server_keyfile parameter when linked with MSVC-built MIT Kerberos. Like the previous commit, no back-patch. http://git.postgresql.org/pg/commitdiff/54aa6ccfc51414b94a2363be6302efb0f911b692
  • Document recipe for testing compatibility with old Perl. Craig Ringer, reviewed by Kyotaro HORIGUCHI and Michael Paquier. http://git.postgresql.org/pg/commitdiff/d61aa6ae655a37d757b68d20ad18a4683c280c14

Correctifs en attente

Pavel Stěhule sent in another revision of a patch to add session server side variables.

Kyotaro HORIGUCHI sent in another revision of a patch to refactor psql's tab completion code for more sanity and add some new tab completions based on this.

Etsuro Fujita sent in another revision of a patch to push down more full joins in the postgres_fdw.

Matheus de Oliveira sent in another revision of a patch to add GRANT/REVOKE ON SCHEMAS to ALTER DEFAULT PRIVILEGES.

Masahiko Sawada sent in another revision of a patch to add quorum commit for multiple synchronous replicas.

Julian Markwort sent in another revision of a patch to add a PGPASSFILE connection option to libpq.

Mithun Cy sent in another revision of a patch to add pg_autoprewarm.

Andrew Borodin sent in two revisions of a patch to make a more concurrency-friendly vacuum of GIN indexes.

Tom Lane sent in a patch to fix the order of tests in findDependentObjects().

Dilip Kumar sent in two more revisions of a patch to push scan keys down to the heap.

Petr Jelínek and Peter Eisentraut traded patches to add logical replication.

Haribabu Kommi sent in another revision of a patch to add a 64-bit (EUI-64) mac address type.

Stephen Frost sent in a patch to fix some compiler warnings.

Emre Hasegeli sent in another revision of a patch to add the customer contains/is contained by operators for the inet data type.

Dilip Kumar sent in another revision of a patch to add parallel bitmap heap scan.

Peter Eisentraut sent in a patch to move collation import to the backend.

Amit Langote sent in a doc patch to clarify a point in ALTER TABLE.

Simon Riggs sent in a patch to make EvalPlanQual() follow the LockTuple() pattern.

Heikki Linnakangas and Michaël Paquier traded patches to add secure random number generation.

Pavel StÄ›hule and Ãlvaro Herrera traded patches to add xmltable().

Etsuro Fujita sent in another revision of a patch to ensure that altering a foreign table invalidats prepared statement execution plans in which it is involved.

Amit Langote sent in another revision of a patch to implement declarative partitioning.

Amit Langote sent in two revisions of a patch to clarify the behavior of ALTER TABLE with parallelism enabled.

Peter Eisentraut sent in another revision of a patch to implement a pg_sequence catalog.

Tomas Vondra sent in another revision of a patch to add two slab-like memory allocators.

Ian Barwick sent in a doc patch to clarify the length of query text displayed in pg_stat_statements.

Stephen Frost sent in another revision of a patch to add support for restrictive RLS policies.

Ashutosh Bapat sent in another revision of a patch to implement partition-wise join for join between (declaratively) partitioned tables.

Thomas Munro sent in another revision of a patch to create a DSA area to provide work space for parallel execution.

Peter Eisentraut sent in another revision of a patch intended to support making DROP FUNCTION able to drop multiple functions per statement.

Mithun Cy sent in a patch to fix a bug in failover at the libpq connect level.

Michaël Paquier sent in another revision of a patch to fix checkpoint skip logic on idle systems by tracking LSN progress.

Amit Kapila sent in two more revisions of a patch to fix an infelicity between parallel execution and prepared statements.

Andreas Karlsson sent in another revision of a patch to reload SSL certificates on SIGHUP.

Pavan Deolasee sent in another revision of a patch to implement WARM.

Thomas Munro sent in a patch to provide the right format string for dsa_pointer to printf-like functions.

Amit Kapila sent in a patch to fix a failed assertion in _hash_splitbucket_guts.

Fabien COELHO sent in another revision of a patch to add backslash-return (well newline really) continuations to all pgbench backslash-commands.

Peter Geoghegan sent in another revision of a patch to add parallel tuplesort.

par N Bougain le mercredi 7 décembre 2016 à 21h56

mardi 29 novembre 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 27 novembre 2016

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en novembre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161127230258.GE21874@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Fix test for subplans in force-parallel mode. We mustn't force parallel mode if the query has any subplans, since ExecSerializePlan doesn't transmit them to workers. Testing top_plan->initPlan is inadequate because (1) there might be initPlans attached to lower plan nodes, and (2) non-initPlan subplans don't work either. There's certainly room for improvement in those restrictions, but for the moment that's what we've got. Amit Kapila, per report from Andreas Seltenreich Discussion: <8737im6pmh.fsf@credativ.de> http://git.postgresql.org/pg/commitdiff/f24cf960d7ae3503e21fcb59dca652575619d9d4
  • Fix optimization for skipping searches for parallel-query hazards. Fix thinko in commit da1c91631: even if the original query was free of parallel hazards, we might introduce such a hazard by adding PARAM_EXEC Param nodes. Adjust is_parallel_safe() so that it will scan the given expression whenever any such nodes have been created. Per report from Andreas Seltenreich. Discussion: <878tse6yvf.fsf@credativ.de> http://git.postgresql.org/pg/commitdiff/4324ade9a6880113b08070305482ace2e8a2617c
  • Fix PGLC_localeconv() to handle errors better. The code was intentionally not very careful about leaking strdup'd strings in case of an error. That was forgivable probably, but it also failed to notice strdup() failures, which could lead to subsequent null-pointer-dereference crashes, since many callers unsurprisingly didn't check for null pointers in the struct lconv fields. An even worse problem is that it could throw error while we were setlocale'd to a non-C locale, causing unwanted behavior in subsequent libc calls. Rewrite to ensure that we cannot throw elog(ERROR) until after we've restored the previous locale settings, or at least attempted to. (I'm sorely tempted to make restore failure be a FATAL error, but will refrain for the moment.) Having done that, it's not much more work to ensure that we clean up strdup'd storage on the way out, too. This code is substantially the same in all supported branches, so back-patch all the way. Michael Paquier and Tom Lane Discussion: <CAB7nPqRMbGqa_mesopcn4MPyTs34eqtVEK7ELYxvvV=oqS00YA@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/a4930e7ca2aa49c13ccba089f5fd2c416d87e96a
  • Fix uninitialized variable. Oversight in a734fd5d1. Michael Paquier http://git.postgresql.org/pg/commitdiff/ae92a9a3806c025653140ee6316a82d55e24b82c
  • Make contrib/test_decoding regression tests safe for CZ locale. A little COLLATE "C" goes a long way. Pavel Stehule, per suggestion from Craig Ringer Discussion: <CAFj8pRA8nJZcozgxN=RMSqMmKuHVOkcGAAKPKdFeiMWGDSUDLA@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/e2a0ee69006bf31f24958b72c93455412cba035c
  • Doc: add a section in Part II concerning RETURNING. There are assorted references to RETURNING in Part II, but nothing that would qualify as an explanation of the feature, which seems like an oversight considering how useful it is. Add something. Noted while looking for a place to point a cross-reference to ... http://git.postgresql.org/pg/commitdiff/1c7861e81b4220364bef75d2445e9c0619f3f3f8
  • Improve handling of "UPDATE ... SET (column_list) = row_constructor". Previously, the right-hand side of a multiple-column assignment, if it wasn't a sub-SELECT, had to be a simple parenthesized expression list, because gram.y was responsible for "bursting" the construct into independent column assignments. This had the minor defect that you couldn't write ROW (though you should be able to, since the standard says this is a row constructor), and the rather larger defect that unlike other uses of row constructors, we would not expand a "foo.*" item into multiple columns. Fix that by changing the RHS to be just "a_expr" in the grammar, leaving it to transformMultiAssignRef to separate the elements of a RowExpr; which it will do only after performing standard transformation of the RowExpr, so that "foo.*" behaves as expected. The key reason we didn't do that before was the hard-wired handling of DEFAULT tokens (SetToDefault nodes). This patch deals with that issue by allowing DEFAULT in any a_expr and having parse analysis throw an error if SetToDefault is found in an unexpected place. That's an improvement anyway since the error can be more specific than just "syntax error". The SQL standard suggests that the RHS could be any a_expr yielding a suitable row value. This patch doesn't really move the goal posts in that respect --- you're still limited to RowExpr or a sub-SELECT --- but it does fix the grammar restriction, so it provides some tangible progress towards a full implementation. And the limitation is now documented by an explicit error message rather than an unhelpful "syntax error". Discussion: <8542.1479742008@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/906bfcad7ba7cb3863fe0e2a7810be8e3cd84fbd
  • Doc: improve documentation about composite-value usage. Create a section specifically for the syntactic rules around whole-row variable usage, such as expansion of "foo.*". This was previously documented only haphazardly, with some critical info buried in unexpected places like xfunc-sql-composite-functions. Per repeated questions in different mailing lists. Discussion: <16288.1479610770@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/e1320266edd7df53c60af10b4c33ab2754278b3e
  • Make sure ALTER TABLE preserves index tablespaces. When rebuilding an existing index, ALTER TABLE correctly kept the physical file in the same tablespace, but it messed up the pg_class entry if the index had been in the database's default tablespace and "default_tablespace" was set to some non-default tablespace. This led to an inaccessible index. Fix by fixing pg_get_indexdef_string() to always include a tablespace clause, whether or not the index is in the default tablespace. The previous behavior was installed in commit 537e92e41, and I think it just wasn't thought through very clearly; certainly the possible effect of default_tablespace wasn't considered. There's some risk in changing the behavior of this function, but there are no other call sites in the core code. Even if it's being used by some third party extension, it's fairly hard to envision a usage that is okay with a tablespace clause being appended some of the time but can't handle it being appended all the time. Back-patch to all supported versions. Code fix by me, investigation and test cases by Michael Paquier. Discussion: <1479294998857-5930602.post@n3.nabble.com> http://git.postgresql.org/pg/commitdiff/bd673e8e864a1987eca8d40c593e857ab8d0a5ba
  • Avoid masking a function parameter name with a local variable name. No actual bug here, but it might confuse readers, so change the name of the local variable. Ashutosh Bapat http://git.postgresql.org/pg/commitdiff/6fa391be4e83139cd134d5ccfc1499809bb8c98c
  • Check that default_tablespace affects ALTER TABLE ADD UNIQUE/PRIMARY KEY. Seems like a good thing to test, considering that we nearly broke it yesterday. Michael Paquier http://git.postgresql.org/pg/commitdiff/4cc6a3f1106034187be3a371e61a02915bb93c11
  • Check for pending trigger events on far end when dropping an FK constraint. When dropping a foreign key constraint with ALTER TABLE DROP CONSTRAINT, we refuse the drop if there are any pending trigger events on the named table; this ensures that we won't remove the pg_trigger row that will be consulted by those events. But we should make the same check for the referenced relation, else we might remove a due-to-be-referenced pg_trigger row for that relation too, resulting in "could not find trigger NNN" or "relation NNN has no triggers" errors at commit. Per bug #14431 from Benjie Gillam. Back-patch to all supported branches. Report: <20161124114911.6530.31200@wrigleys.postgresql.org> http://git.postgresql.org/pg/commitdiff/4e026b32d4024b03856b4981b26c747b7fef7afb
  • Mark a query's topmost Paths parallel-unsafe if they will have initPlans. Andreas Seltenreich found another case where we were being too optimistic about allowing a plan to be considered parallelizable despite it containing initPlans. It seems like the real issue here is that if we know we are going to tack initPlans onto the topmost Plan node for a subquery, we had better mark that subquery's result Paths as not-parallel-safe. That fixes this problem and allows reversion of a kluge (added in commit 7b67a0a49 and extended in f24cf960d) to not trust the parallel_safe flag at top level. Discussion: <874m2w4k5d.fsf@ex.ansel.ydns.eu> http://git.postgresql.org/pg/commitdiff/ab77a5a4561fad847af4a101a29c922c66449870
  • Bring some clarity to the defaults for the xxx_flush_after parameters. Instead of confusingly stating platform-dependent defaults for these parameters in the comments in postgresql.conf.sample (with the main entry being a lie on Linux), teach initdb to install the correct platform-dependent value in postgresql.conf, similarly to the way we handle other platform-dependent defaults. This won't do anything for existing 9.6 installations, but since it's effectively only a documentation improvement, that seems OK. Since this requires initdb to have access to the default values, move the #define's for those to pg_config_manual.h; the original placement in bufmgr.h is unworkable because that file can't be included by frontend programs. Adjust the default value for wal_writer_flush_after so that it is 1MB regardless of XLOG_BLCKSZ, conforming to what is stated in both the SGML docs and postgresql.conf. (We could alternatively make it scale with XLOG_BLCKSZ, but I'm not sure I see the point.) Copy-edit related SGML documentation. Fabien Coelho and Tom Lane, per a gripe from Tomas Vondra. Discussion: <30ebc6e3-8358-09cf-44a8-578252938424@2ndquadrant.com> http://git.postgresql.org/pg/commitdiff/dbdfd114f34443f1e4ad16ce2721f9817d3b3d80
  • Fix test about ignoring extension dependencies during extension scripts. Commit 08dd23cec introduced an exception to the rule that extension member objects can only be dropped as part of dropping the whole extension, intending to allow such drops while running the extension's own creation or update scripts. However, the exception was only applied at the outermost recursion level, because it was modeled on a pre-existing check to ignore dependencies on objects listed in pendingObjects. Bug #14434 from Philippe Beaudoin shows that this is inadequate: in some cases we can reach an extension member object by recursion from another one. (The bug concerns the serial-sequence case; I'm not sure if there are other cases, but there might well be.) To fix, revert 08dd23cec's changes to findDependentObjects() and instead apply the creating_extension exception regardless of stack level. Having seen this example, I'm a bit suspicious that the pendingObjects logic is also wrong and such cases should likewise be allowed at any recursion level. However, changing that would interact in subtle ways with the recursion logic (at least it would need to be moved to after the recursing-from check). Given that the code's been like that a long time, I'll refrain from touching it without a clear example showing it's wrong. Back-patch to all active branches. In HEAD and 9.6, where suitable test infrastructure exists, add a regression test case based on the bug report. Report: <20161125151448.6529.33039@wrigleys.postgresql.org> Discussion: <13224.1480177514@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/182db070403fb33566da156a3a77cbcda16583b4

Robert Haas pushed:

  • autovacuum: Drop orphan temp tables more quickly but with more caution. Previously, we only dropped an orphan temp table when it became old enough to threaten wraparound; instead, doing it immediately. The only value of waiting is that someone might be able to examine the contents of the orphan temp table for forensic purposes, but it's pretty difficult to actually do that and few users will wish to do so. On the flip side, not performing the drop immediately generates log spam and bloats pg_class. In addition, per a report from Grigory Smolkin, if a temporary schema contains a very large number of temporary tables, a backend attempting to clear the temporary schema might fail due to lock table exhaustion. It's helpful for autovacuum to clean up after such cases, and we don't want it to wait for wraparound to threaten before doing so. To prevent autovacuum from failing in the same manner as a backend trying to drop an entire temp schema, remove orphan temp tables in batches of 50, committing after each batch, so that we don't accumulate an unbounded number of locks. If a drop fails, retry other orphan tables that need to be dropped up to 10 times before giving up. With this system, if a backend does fail to clean a temporary schema due to lock table exhaustion, autovacuum should hopefully put things right the next time it processes the database. Discussion: CAB7nPqSbYT6dRwsXVgiKmBdL_ARemfDZMPA+RPeC_ge0GK70hA@mail.gmail.com Michael Paquier, with a bunch of comment changes by me. http://git.postgresql.org/pg/commitdiff/a734fd5d1c309cc553b7c8c79fba96218af090f7
  • Support condition variables. Condition variables provide a flexible way to sleep until a cooperating process causes an arbitrary condition to become true. In simple cases, this can be accomplished with a WaitLatch/ResetLatch loop; the cooperating process can call SetLatch after performing work that might cause the condition to be satisfied, and the waiting process can recheck the condition each time. However, if the process performing the work doesn't have an easy way to identify which processes might be waiting, this doesn't work, because it can't identify which latches to set. Condition variables solve that problem by internally maintaining a list of waiters; a process that may have caused some waiter's condition to be satisfied must "signal" or "broadcast" on the condition variable. Robert Haas and Thomas Munro http://git.postgresql.org/pg/commitdiff/e8ac886c24776295dd9b025386a821061da8e4d1
  • Code review for commit 274bb2b3857cc987cfa21d14775cae9b0dababa5. Avoid memory leak in conninfo_uri_parse_options. Use the current host rather than the comma-separated list of host names when the host name is needed for GSS, SSPI, or SSL authentication. Document the way connect_timeout interacts with multiple host specifications. Takayuki Tsunakawa http://git.postgresql.org/pg/commitdiff/9a1d0af4ad2cbd419115b453d811c141b80d872b
  • Remove barrier.h. A new thing also called a "barrier" is proposed, but whether we decide to take that patch or not, this file seems to have outlived its usefulness. Thomas Munro http://git.postgresql.org/pg/commitdiff/e343dfa42bff35983c582da3916b205763aeac90
  • Mark IsPostmasterEnvironment and IsBackgroundWorker as PGDLLIMPORT. Per request from Craig Ringer. http://git.postgresql.org/pg/commitdiff/273270593f42bdbd5422923eb70fbd0fb0f65bf0

Ãlvaro Herrera pushed:

Magnus Hagander pushed:

Correctifs en attente

Craig Ringer sent in two revisions of a patch to document that test must pass with Perl 5.8.0 and show how to ensure that they do.

Michaël Paquier sent in another revision of a patch to fix the checkpoint skip logic on idle systems by tracking LSN progress.

Corey Huinker sent in a patch to ensure that dblink knows about valid PostgreSQL FDW options.

Thomas Munro sent in another revision of a patch to add delta relations in AFTER triggers, complete with exposure to PL/pgsql and PL/PythonU.

KaiGai Kohei sent in another revision of a patch to enable passing LIMIT to FDWs AND CUSTOMSCANS.

Daniel Vérité sent in two more revisions of a patch to improve psql hooks for variables.

Amit Kapila sent in a patch to prevent cascading standbys from getting stuck.

Etsuro Fujita sent in three more revisions of a patch to push down more full joins to the PostgreSQL FDW.

Craig Ringer sent in a patch to add logical decoding on standbys.

Thomas Munro sent in another revision of a patch to measure replay lag.

Matheus de Oliveira sent in a patch to extend ALTER DEFAULT PRIVILEGES to include schemas.

Thomas Munro sent in another revision of a patch to add barriers for synchronizing cooperating processes.

Peter Geoghegan sent in another revision of a patch to add amcheck, a B-Tree integrity checking tool.

Mithun Cy sent in two more revisions of a patch to implement failover on the libpq connect level.

Michaël Paquier sent in another revision of a patch to forbid the use of LF and CR characters in database and role names.

Amit Langote sent in two more revisions of a patch to add declarative partitioning.

Erik Rijkers sent in a patch to fix the CREATE SUBSCRIPTION documents, which broke the oldhtml target.

Andres Freund sent in a patch to fix a performance degradation in Bitmapscan.

Amul Sul sent in another revision of a patch to add a pg_background contrib extension.

Amit Kapila sent in another revision of a patch to make hash indexes concurrency-safe.

Ãlvaro Herrera and Pavel StÄ›hule traded patches to add xmltable().

Thomas Munro sent in two more revisions of a patch to add dynamic shared memory areas (DSAs).

Thomas Munro sent in two more revisions of a patch to add a DSA for the parallel execution.

Karl O. Pinc and Gilles Darold traded patches to implement pg_current_logfile().

Rushabh Lathia sent in another revision of a patch to implement Gather Merge.

Andreas Karlsson sent in a WIP patch to make PQHost() return the value of the host parameter rather than the hostaddr.

Amit Kapila sent in two revisions of a patch to allow safe subquery with parallel workers.

Haribabu Kommi sent in another revision of a patch to add an 8-byte MAC address type called macaddr8 which conforms to EUI-64.

Pavel Stěhule sent in a PoC patch to add session server side variables.

Ming Li sent in a patch to fix a pg_class refcache leak when the meta tuple in pg_class is invalid.

Amit Kapila sent in another revision of a patch to add parallel index scans.

Michaël Paquier sent in another revision of a patch to add quorum commit for multiple synchronous replication.

Artur Zakirov sent in a patch to fix a bug in which caused a pg_event_trigger_ddl_commands() error with ALTER TEXT SEARCH CONFIGURATION.

par N Bougain le mardi 29 novembre 2016 à 02h08

samedi 26 novembre 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 20 novembre 2016

Le nouveau magazine PostgreSQL est disponible ! http://www.pgmag.org/

La PGconf.ASIA 2016 aura lieu les 1, 2 & 3 décembre à Tokyo (Japon). Les inscriptions sont ouvertes. Il y aura une UnConference le 1er décembre avec une inscription séparée et requise : http://www.pgconf.asia/EN/registration/ http://www.pgconf.asia/EN/day-0/

La PGConf India 2017 aura lieu les 2 & 3 mars 2017 à Bengalore (État du Karnataka en Inde). Les propositions sont attendues sur <papers AT pgconf DOT in> jusqu'au 31 décembre 2016 : http://pgconf.in/

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en novembre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161120225904.GB23469@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Ãlvaro Herrera pushed:

  • Fix duplication in ALTER MATERIALIZE VIEW synopsis. Commit 3c4cf080879b should have removed SET TABLESPACE from the synopsis of ALTER MATERIALIZE VIEW as a possible "action" when it added a separate line for it in the main command listing, but failed to. Repair. Backpatch to 9.4, like the aforementioned commit. http://git.postgresql.org/pg/commitdiff/8ce4f597abc530b3b59bcf3a3964f31e50054bcd
  • Avoid pin scan for replay of XLOG_BTREE_VACUUM in all cases Replay of XLOG_BTREE_VACUUM during Hot Standby was previously thought to require complex interlocking that matched the requirements on the master. This required an O(N) operation that became a significant problem with large indexes, causing replication delays of seconds or in some cases minutes while the XLOG_BTREE_VACUUM was replayed. This commit skips the “pin scan†that was previously required, by observing in detail when and how it is safe to do so, with full documentation. The pin scan is skipped only in replay; the VACUUM code path on master is not touched here. No tests included. Manual tests using an additional patch to view WAL records and their timing have shown the change in WAL records and their handling has successfully reduced replication delay. This is a back-patch of commits 687f2cd7a015, 3e4b7d87988f, b60284261375 by Simon Riggs, to branches 9.4 and 9.5. No further backpatch is possible because this depends on catalog scans being MVCC. I (Ãlvaro) additionally updated a slight problem in the README, which explains why this touches the 9.6 and master branches. http://git.postgresql.org/pg/commitdiff/f65b94f63962e9f7e144a469bc1750286ddaee27

Peter Eisentraut pushed:

  • Allow individual TAP tests to be run via PROVE_TESTS. Add a new optional Makefile variable PROVE_TESTS that, if passed as a space-separated list of paths relative to the Makefile invoking $(prove_check) or $(prove_installcheck), runs just those tests instead of t/*.pl . From: Craig Ringer <craig@2ndquadrant.com> Reviewed-by: Michael Paquier <michael.paquier@gmail.com> http://git.postgresql.org/pg/commitdiff/9ca7b0bf016364c74d38f66c7050be915bfea908
  • Build HTML documentation using XSLT stylesheets by default. The old DSSSL build is still available for a while using the make target "oldhtml". http://git.postgresql.org/pg/commitdiff/e36ddab11735052841b4eff96642187ec9a8a7bc
  • doc: Further XSLT HTML build performance optimization. Cut out some expensive stuff from the HTML head element that we don't really need. This was previously discussed as part of e8306745e3504c642f7abad411139d5630e29fac, but ended up separate because it changes the output contents slightly. http://git.postgresql.org/pg/commitdiff/380895f2deb18ed9e7a8be69961af2ed221ba9d3
  • Add pg_sequences view. Like pg_tables, pg_views, and others, this view contains information about sequences in a way that is independent of the system catalog layout but more comprehensive than the information schema. To help implement the view, add a new internal function pg_sequence_last_value() to return the last value of a sequence. This is kept separate from pg_sequence_parameters() to separate querying run-time state from catalog-like information. Reviewed-by: Andreas Karlsson <andreas@proxel.se> http://git.postgresql.org/pg/commitdiff/67dc4ccbb2e1c27da823eced66d9217a5652cbb0

Magnus Hagander pushed:

Andres Freund pushed:

Robert Haas pushed:

  • pgbench: Increase maximum size of log filename from 64 to MAXPGPATH. Commit 41124a91e61fc6d9681c1e8b15ba30494e84d643 allowed the transaction log file prefix to be changed but left in place the existing 64-character limit on the total length of a log file name. It's possible that could be inconvenient for somebody, so increase the limit to MAXPGPATH, which ought to be enough for anybody. Per a suggestion from Tom Lane. http://git.postgresql.org/pg/commitdiff/56eba9b8a1120c861868dd3d86d927a9e3182880
  • Fix broken statement in UCS_to_most.pl. This has been wrong for a very long time, and it's puzzling to me how it ever worked for anyone. Kyotaro Horiguchi http://git.postgresql.org/pg/commitdiff/00c6d8077f39191a6f61a847ce7d55073d8f5a6f
  • Limit the number of number of tapes used for a sort to 501. Gigantic numbers of tapes don't work out well. Original patch by Peter Geoghegan; comments entirely rewritten by me. http://git.postgresql.org/pg/commitdiff/fc19c1801bd2dbee1043b0c0b62e07747d30ea1c
  • Reserve zero as an invalid DSM handle. Previously, the handle for the control segment could not be zero, but some other DSM segment could potentially have a handle value of zero. However, that means that if someone wanted to store a dsm_handle that might or might not be valid, they would need a separate boolean to keep track of whether the associated value is legal. That's annoying, so change things so that no DSM segment can ever have a handle of 0 - or as we call it here, DSM_HANDLE_INVALID. Thomas Munro. This was submitted as part of a much larger patch to add an malloc-like allocator for dynamic shared memory, but this part seems like a good idea independently of the rest of the patch. http://git.postgresql.org/pg/commitdiff/b40b4dd9e10ea701c8d47ccba9407fc32ed384e5
  • Remove or reduce verbosity of some debug messages. The debug messages that merely print StartTransactionCommand, CommitTransactionCommand, ProcessUtilty, or ProcessQuery with no additional details seem to be useless. Get rid of them. The transaction status messages produced by ShowTransactionState are occasionally useful, but they are extremely verbose, producing multiple lines of log output every time they fire, which can happens multiple times per transaction. So, reduce the level to DEBUG5; avoid emitting an extra line just to explain which debug point is at issue; and tighten up the rest of the message so it doesn't use quite so much horizontal space. With these changes, it's possible to run a somewhat busy system with a log level even as high as DEBUG4, whereas previously anything above DEBUG2 would flood the log with output that probably wasn't really all that useful. http://git.postgresql.org/pg/commitdiff/a43f1939d5dcd02f4df1604a68392332168e4be0

Tom Lane pushed:

  • Account for catalog snapshot in PGXACT->xmin updates. The CatalogSnapshot was not plugged into SnapshotResetXmin()'s accounting for whether MyPgXact->xmin could be cleared or advanced. In normal transactions this was masked by the fact that the transaction snapshot would be older, but during backend startup and certain utility commands it was possible to re-use the CatalogSnapshot after MyPgXact->xmin had been cleared, meaning that recently-deleted rows could be pruned even though this snapshot could still see them, causing unexpected catalog lookup failures. This effect appears to be the explanation for a recent failure on buildfarm member piculet. To fix, add the CatalogSnapshot to the RegisteredSnapshots heap whenever it is valid. In the previous logic, it was possible for the CatalogSnapshot to remain valid across waits for client input, but with this change that would mean it delays advance of global xmin in cases where it did not before. To avoid possibly causing new table-bloat problems with clients that sit idle for long intervals, add code to invalidate the CatalogSnapshot before waiting for client input. (When the backend is busy, it's unlikely that the CatalogSnapshot would be the oldest snap for very long, so we don't worry about forcing early invalidation of it otherwise.) In passing, remove the CatalogSnapshotStale flag in favor of using "CatalogSnapshot != NULL" to represent validity, as we do for the other special snapshots in snapmgr.c. And improve some obsolete comments. No regression test because I don't know a deterministic way to cause this failure. But the stress test shown in the original discussion provokes "cache lookup failed for relation 1255" within a few dozen seconds for me. Back-patch to 9.4 where MVCC catalog scans were introduced. (Note: it's quite easy to produce similar failures with the same test case in branches before 9.4. But MVCC catalog scans were supposed to fix that.) Discussion: <16447.1478818294@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/ffaa44cb559db332baeee7d25dedd74a61974203
  • Allow DOS-style line endings in ~/.pgpass files. On Windows, libc will mask \r\n line endings for us, since we read the password file in text mode. But that doesn't happen on Unix. People who share password files across both systems might have \r\n line endings in a file they use on Unix, so as a convenience, ignore trailing \r. Per gripe from Josh Berkus. In passing, put the existing check for empty line somewhere where it's actually useful, ie after stripping the newline not before. Vik Fearing, adjusted a bit by me Discussion: <0de37763-5843-b2cc-855e-5d0e5df25807@agliodbs.com> http://git.postgresql.org/pg/commitdiff/0a7481930c788e9d74a154aac0c8b401fc6a81f9
  • Check that result tupdesc has exactly 1 column in return_next scalar case. This should always be true, but since we're relying on a tuple descriptor passed from outside pltcl itself, let's check. Per a gripe from Coverity. http://git.postgresql.org/pg/commitdiff/4ecd1974377ffb4d6d72874ba14fcd23965b1792
  • Re-pgindent src/bin/pg_dump/* Cleanup for recent patches --- it's not much change, but I got annoyed while re-indenting the view-rule fix I'm working on. http://git.postgresql.org/pg/commitdiff/fcf70e0dbca1432959be5f3557acd546d639c9ba
  • Improve pg_dump/pg_restore --create --if-exists logic. Teach it not to complain if the dropStmt attached to an archive entry is actually spelled CREATE OR REPLACE VIEW, since that will happen due to an upcoming bug fix. Also, if it doesn't recognize a dropStmt, have it print a WARNING and then emit the dropStmt unmodified. That seems like a much saner behavior than Assert'ing or dumping core due to a null-pointer dereference, which is what would happen before :-(. Back-patch to 9.4 where this option was introduced. Discussion: <19092.1479325184@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/ac888986fc656c0badb0a4918b3581e2ebcb154a
  • Fix pg_dump's handling of circular dependencies in views. pg_dump's traditional solution for breaking a circular dependency involving a view was to create the view with CREATE TABLE and then later issue CREATE RULE "_RETURN" ... to convert the table to a view, relying on the backend's very very ancient code that supports making views that way. We've wanted to get rid of that kluge for a long time, but the thing that finally motivates doing something about it is the recognition that this method fails with the --clean option, because it leads to issuing DROP RULE "_RETURN" followed by DROP TABLE --- and the backend won't let you drop a view's _RETURN rule. Instead, let's break circular dependencies by initially creating the view using CREATE VIEW AS SELECT NULL::columntype AS columnname, ... (so that it has the right column names and types to support external references, but no dependencies beyond the column data types), and then later dumping the ON SELECT rule using the spelling CREATE OR REPLACE VIEW. This method wasn't available when this code was originally written, but it's been possible since PG 7.3, so it seems fine to start relying on it now. To solve the --clean problem, make the dropStmt for an ON SELECT rule be CREATE OR REPLACE VIEW with the same dummy target list as above. In this way, during the DROP phase, we first reduce the view to have no extra dependencies, and then we can drop it entirely when we've gotten rid of whatever had a circular dependency on it. (Note: this should work adequately well with the --if-exists option, since the CREATE OR REPLACE VIEW will go through whether the view exists or not. It could fail if the view exists with a conflicting column set, but we don't really support --clean against a non-matching database anyway.) This allows cleaning up some other kluges inside pg_dump, notably that we don't need a notion of reloptions attached to a rule anymore. Although this is a bug fix, commit to HEAD only for now. The problem's existed for a long time and we've had relatively few complaints, so it doesn't really seem worth taking risks to fix it in the back branches. We might revisit that choice if no problems emerge. Discussion: <19092.1479325184@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/d8c05aff56be92dda889abc89e3f3921d1c29c30
  • Code review for GUC serialization/deserialization code. The serialization code dumped core for a string-valued GUC whose value is NULL, which is a legal state. The infrastructure isn't capable of transmitting that state exactly, but fortunately, transmitting an empty string instead should be close enough (compare, eg, commit e45e990e4). The code potentially underestimated the space required to format a real-valued variable, both because it made an unwarranted assumption that %g output would never be longer than %e output, and because it didn't count right even for %e format. In practice this would pretty much always be masked by overestimates for other variables, but it's still wrong. Also fix boundary-case error in read_gucstate, incorrect handling of the case where guc_sourcefile is non-NULL but zero length (not clear that can happen, but if it did, this code would get totally confused), and confusingly useless check for a NULL result from read_gucstate. Andreas Seltenreich discovered the core dump; other issues noted while reading nearby code. Back-patch to 9.5 where this code was introduced. Michael Paquier and Tom Lane Discussion: <871sy78wno.fsf@credativ.de> http://git.postgresql.org/pg/commitdiff/13671b4b22ae4bd345c62e7c0b41d717b8a2e19b
  • Fix latent costing error in create_merge_append_path. create_merge_append_path should use the path rowcount it just computed, not rel->tuples, for costing purposes. Those numbers should always be the same at present, but if we ever support parameterized MergeAppend paths (a case this function is otherwise prepared for), the former would be right and the latter wrong. No need for back-patch since the problem is only latent. Ashutosh Bapat Discussion: <CAFjFpRek+cLCnTo24youuGtsq4zRphEB8EUUPjDxZjnL4n4HYQ@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/0832f2db68cc43524a240db47d0428cc9525723e
  • Prevent multicolumn expansion of "foo.*" in an UPDATE source expression. Because we use transformTargetList() for UPDATE as well as SELECT tlists, the code accidentally tried to expand a "*" reference into several columns. This is nonsensical, because the UPDATE syntax provides exactly one target column to put the value into. The immediate result was that transformUpdateTargetList() got confused and reported "UPDATE target count mismatch --- internal error". It seems better to treat such a reference as a plain whole-row variable, as it would be in other contexts. (This could produce useful results when the target column is of composite type.) Fix by tweaking transformTargetList() to perform *-expansion only conditionally, depending on its exprKind parameter. Back-patch to 9.3. The problem exists further back, but a fix would be much more invasive before that, because transformTargetList() wasn't told what kind of list it was working on. Doesn't seem worth the trouble given the lack of field reports. (I only noticed it because I was checking the code while trying to improve the documentation about how we handle "foo.*".) Discussion: <4308.1479595330@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/c5f365f3ab21a345b7a109bc0411895536e9fa69

Stephen Frost pushed:

  • Clean up pg_dump tests, re-enable BLOB testing Add a loop to check that each test covers all of the pg_dump runs. We (I) had been a bit sloppy when adding new runs and not making sure to mark if they should be under like or unlike for each test, this loop makes sure that the test system will complain if any are forgotten in the future. The loop also correctly handles the 'catch all' cases, which are used to avoid running unnecessary specific checks when a single catch-all can be done (eg: a no-acl run should not have any GRANT commands). Also, re-enable the testing of blobs, but use lo_from_bytea() instead of trying to be cute and writing out to a file and then reading it back in with psql, which proved to be difficult for some buildfarm members. This allows us to add support for testing the --no-blobs option which will be getting added shortly, provided the buildfarm doesn't blow up on this. http://git.postgresql.org/pg/commitdiff/8f91f323b4feef0371cd3db51be3007e44abd5e8

Correctifs en attente

Tomas Vondra sent in another revision of a patch to add two slab-like memory allocators.

Laurenz Albe sent in another revision of a patch to ensure that writes are synced to disk.

Michaël Paquier and Robert Haas traded patches to add SCRAM auth with provisions for more types.

Takayuki Tsunakawa sent in another revision of a patch to fix a situation where a cascaded standby gets stuck and can't catch up.

Kyotaro HORIGUCHI sent in another revision of a patch to add asynchronous execution.

Kyotaro HORIGUCHI sent in another revision of a patch to rework psql's tab completion system.

Magnus Hagander sent in a patch to add a capability to jsonb_delete to accept an array of keys rather than a singleton one.

Kuntal Ghosh sent in another revision of a patch to add a WAL consistency check facility.

Thomas Munro sent in another revision of a patch to enable SERIALIZABLE isolation mode on standby servers.

Thomas Munro sent in another revision of a patch to create dynamic shared memory areas.

Masahiko Sawada sent in another revision of a patch to enable quorum commit for multiple synchronous standbys.

Craig Ringer sent in two more revisions of a patch to enable logical decoding timeline following.

Andrew Borodin sent in another revision of a patch to implement lazy GiST indexing.

Mark Kirkwood sent in another revision of a patch to switch to the CMake build system.

Pavel Stěhule sent in three more revisions of a patch to implement \setfileref in psql.

Etsuro Fujita and Ashutosh Bapat traded patches to push down more full joins to FDWs.

Amit Langote sent in three more revisions of a patch to implement declarative partitioning.

Magnus Hagander sent in a patch to log "snapshot too old" conditions.

Laurenz Albe and Tobias Bussmann traded patches to enable parallel prepared statement execution.

Karl O. Pinc and Gilles Darold traded patches to implement pg_current_logfile().

Daniel Vérité sent in another revision of a patch to add capabilities to psql hooks for variables.

Ramon Lawrence sent in another revision of a patch to implement Gather Merge.

Christian Ullrich sent in another revision of a patch to add putenv support for msvcrt from Visual Studio 2013.

Rahila Syed sent in a patch to fix some issues that can result in an invalid collation error.

Haribabu Kommi sent in another revision of a patch to add a pg_hba_file_settings view.

Michaël Paquier sent in two more revisions of a patch to fix cleanup of unlogged tables.

Claudio Freire sent in another revision of a patch to enable vacuum to use more than 1GB of work mem.

Craig Ringer sent in a patch to document that perl 5.8.4 or higher is required for TAP tests.

Craig Ringer sent in a patch to allow walsender to exit on conflict with recovery.

Etsuro Fujita sent in a patch to push down more UPDATEs/DELETEs in the postgres_fdw.

Kyotaro HORIGUCHI sent in two approaches to fix an issue where WALs were being recycled too aggressively in synchronous replication. The first injects a pause. The second blocks WAL insertion until sync replication reaches the first surviving segments.

Michaël Paquier sent in two more revisions of a patch to clean up orphan temp tables with autovacuum.

Mithun Cy sent in two more revisions of a patch to implement failover on the libpq connect level.

Ashutosh Bapat sent in a patch to fix estimated tuple counts for appendrel.

Pavel Stěhule sent in another revision of a patch to implement xmltable().

Stephen Frost sent in a doc patch to clarify that blobs are considered data and not schema for pg_dump purposes.

John Gorman sent in a patch to speed up dynamic shared memory hash tables.

Gilles Darold sent in a patch to add psql tab completion for ALTER DEFAULT PRIVILEGES.

Petr Jelínek sent in another revision of a patch to add logical replication.

par N Bougain le samedi 26 novembre 2016 à 16h15

vendredi 4 novembre 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 30 octobre 2016

Versions correctives 9.6.1, 9.5.5, 9.4.10, 9.3.15, 9.2.19 et 9.1.24 disponibles. Mettez à jour dès que possible. La 9.1.24 est la dernière publication de la série 9.1 : https://www.postgresql.org/about/news/1712/

Offres d'emplois autour de PostgreSQL en octobre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161031023727.GE23627@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Don't throw serialization errors for self-conflicts in INSERT ON CONFLICT. A transaction that conflicts against itself, for example INSERT INTO t(pk) VALUES (1),(1) ON CONFLICT DO NOTHING; should behave the same regardless of isolation level. It certainly shouldn't throw a serialization error, as retrying will not help. We got this wrong due to the ON CONFLICT logic not considering the case, as reported by Jason Dusek. Core of this patch is by Peter Geoghegan (based on an earlier patch by Thomas Munro), though I didn't take his proposed code refactoring for fear that it might have unexpected side-effects. Test cases by Thomas Munro and myself. Report: <CAO3NbwOycQjt2Oqy2VW-eLTq2M5uGMyHnGm=RNga4mjqcYD7gQ@mail.gmail.com> Related-Discussion: <57EE93C8.8080504@postgrespro.ru> http://git.postgresql.org/pg/commitdiff/a6c0a5b6e8a9498540c6a7bb1b6d68958acc9bc6
  • Avoid testing tuple visibility without buffer lock. INSERT ... ON CONFLICT (specifically ExecCheckHeapTupleVisible) contains another example of this unsafe coding practice. It is much harder to get a failure out of it than the case fixed in commit 6292c2339, because in most scenarios any hint bits that could be set would have already been set earlier in the command. However, Konstantin Knizhnik reported a failure with a custom transaction manager, and it's clearly possible to get a failure via a race condition in async-commit mode. For lack of a reproducible example, no regression test case in this commit. I did some testing with Asserts added to tqual.c's functions, and can say that running "make check-world" exposed these two bugs and no others. The Asserts are messy enough that I've not added them to the code for now. Report: <57EE93C8.8080504@postgrespro.ru> Related-Discussion: <CAO3NbwOycQjt2Oqy2VW-eLTq2M5uGMyHnGm=RNga4mjqcYD7gQ@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/8f1fb7d621b0e6bd2eb0ba2ac9634c5b5a03564b
  • Release notes for 9.6.1, 9.5.5, 9.4.10, 9.3.15, 9.2.19, 9.1.24. http://git.postgresql.org/pg/commitdiff/7d80417d3dfc88b0c03b5c08a18b29f9d430e217
  • Update release notes for last-minute commit timestamp fix. http://git.postgresql.org/pg/commitdiff/30a6f98ed8a2223ef876197b5dea0d94e5807b51
  • Suppress unused-variable warning in non-assert builds. Introduced in commit 7012b132d. Kyotaro Horiguchi http://git.postgresql.org/pg/commitdiff/8529686ccbb9f1c2fe350920ad324c223135a957
  • Doc: improve documentation about inheritance. Clarify documentation about inheritance of check constraints, in particular mentioning the NO INHERIT option, which didn't exist when this text was written. Document that in an inherited query, the applicable row security policies are those of the explicitly-named table, not its children. This is the intended behavior (per off-list discussion with Stephen Frost), and there are regression tests for it, but it wasn't documented anywhere user-facing as far as I could find. Do a bit of wordsmithing on the description of inherited access-privilege checks. Back-patch to 9.5 where RLS was added. http://git.postgresql.org/pg/commitdiff/162477a63d3c0fd1c31197717140a88077a8d0aa
  • Fix not-HAVE_SYMLINK code in zic.c. I broke this in commit f3094920a. Apparently it's dead code anyway, at least as far as our buildfarm is concerned (and the upstream IANA code doesn't worry at all about symlink() not being present). But as long as the rest of our code is willing to guard against not having symlink(), this should too. Noted while investigating a tangentially-related complaint from Sandeep Thakkar. Back-patch to keep branches in sync. http://git.postgresql.org/pg/commitdiff/19b2094d96807e43d29687b3860e8fffb9da61b4
  • Fix incorrect trigger-property updating in ALTER CONSTRAINT. The code to change the deferrability properties of a foreign-key constraint updated all the associated triggers to match; but a moment's examination of the code that creates those triggers in the first place shows that only some of them should track the constraint's deferrability properties. This leads to odd failures in subsequent exercise of the foreign key, as the triggers are fired at the wrong times. Fix that, and add a regression test comparing the trigger properties produced by ALTER CONSTRAINT with those you get by creating the constraint as-intended to begin with. Per report from James Parks. Back-patch to 9.4 where this ALTER functionality was introduced. Report: <CAJ3Xv+jzJ8iNNUcp4RKW8b6Qp1xVAxHwSXVpjBNygjKxcVuE9w@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/a522fc3d806ca6ebe3f66e55b3f8cecb85116711
  • Improve speed of aggregates that use array_append as transition function. In the previous coding, if an aggregate's transition function returned an expanded array, nodeAgg.c and nodeWindowAgg.c would always copy it and thus force it into the flat representation. This led to ping-ponging between flat and expanded formats, which costs a lot. For an aggregate using array_append as transition function, I measured about a 15X slowdown compared to the pre-9.5 code, when working on simple int[] arrays. Of course, the old code was already O(N^2) in this usage due to copying flat arrays all the time, but it wasn't quite this inefficient. To fix, teach nodeAgg.c and nodeWindowAgg.c to allow expanded transition values without copying, so long as the transition function takes care to return the transition value already properly parented under the aggcontext. That puts a bit of extra responsibility on the transition function, but doing it this way allows us to not need any extra logic in the fast path of advance_transition_function (ie, with a pass-by-value transition value, or with a modified-in-place pass-by-reference value). We already know that that's a hot spot so I'm loath to add any cycles at all there. Also, while only array_append currently knows how to follow this convention, this solution allows other transition functions to opt-in without needing to have a whitelist in the core aggregation code. (The reason we would need a whitelist is that currently, if you pass a R/W expanded-object pointer to an arbitrary function, it's allowed to do anything with it including deleting it; that breaks the core agg code's assumption that it should free discarded values. Returning a value under aggcontext is the transition function's signal that it knows it is an aggregate transition function and will play nice. Possibly the API rules for expanded objects should be refined, but that would not be a back-patchable change.) With this fix, an aggregate using array_append is no longer O(N^2), so it's much faster than pre-9.5 code rather than much slower. It's still a bit slower than the bespoke infrastructure for array_agg, but the differential seems to be only about 10%-20% rather than orders of magnitude. Discussion: <6315.1477677885@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/9a00f03e479c2d4911eed5b4bd1994315d409938
  • Fix bogus tree-flattening logic in QTNTernary(). QTNTernary() contains logic to flatten, eg, '(a & b) & c' into 'a & b & c', which is all well and good, but it tries to do that to NOT nodes as well, so that '!!a' gets changed to '!a'. Explicitly restrict the conversion to be done only on AND and OR nodes, and add a test case illustrating the bug. In passing, provide some comments for the sadly naked functions in tsquery_util.c, and simplify some baroque logic in QTNFree(), which I think may have been leaking some items it intended to free. Noted while investigating a complaint from Andreas Seltenreich. Back-patch to all supported versions. http://git.postgresql.org/pg/commitdiff/24ebc444c61306f50777f674544e8559e765ad81
  • Fix nasty performance problem in tsquery_rewrite(). tsquery_rewrite() tries to find matches to subsets of AND/OR conditions; for example, in the query 'a | b | c' the substitution subquery 'a | c' should match and lead to replacement of the first and third items. That's fine, but the matching algorithm apparently takes about O(2^N) for an N-clause query (I say "apparently" because the code is also both unintelligible and uncommented). We could probably do better than that even without any extra assumptions --- but actually, we know that the subclauses are sorted, indeed are depending on that elsewhere in this very same function. So we can just scan the two lists a single time to detect matches, as though we were doing a merge join. Also do a re-flattening call (QTNTernary()) in tsquery_rewrite_query, just to make sure that the tree fits the expectations of the next search cycle. I didn't try to devise a test case for this, but I'm pretty sure that the oversight could have led to failure to match in some cases where a match would be expected. Improve comments, and also stick a CHECK_FOR_INTERRUPTS into dofindsubquery, just in case it's still too slow for somebody. Per report from Andreas Seltenreich. Back-patch to all supported branches. Discussion: <8760oasf2y.fsf@credativ.de> http://git.postgresql.org/pg/commitdiff/5ec81aceec8cc5a0edcc4609ee8edbc43a47e878

Álvaro Herrera pushed:

Magnus Hagander pushed:

Robert Haas pushed:

  • postgres_fdw: Try again to stabilize aggregate pushdown regression tests. A query that only aggregates one row isn't a great argument for pushdown, and buildfarm member brolga decides against it. Adjust the query a bit in the hopes of getting remote aggregation to win consistently. Jeevan Chalke, per suggestion from Tom Lane http://git.postgresql.org/pg/commitdiff/f5d6bce63ceb3c59a964814bb0df5a0648e750e5
  • Fix possible pg_basebackup failure on standby with "include WAL". If a restartpoint flushed no dirty buffers, it could fail to update the minimum recovery point, leading to a minimum recovery point prior to the starting REDO location. perform_base_backup() would interpret that as meaning that no WAL files at all needed to be included in the backup, failing an internal sanity check. To fix, have restartpoints always update the minimum recovery point to just after the checkpoint record itself, so that the file (or files) containing the checkpoint record will always be included in the backup. Code by Amit Kapila, per a design suggestion by me, with some additional work on the code comment by me. Test case by Michael Paquier. Report by Kyotaro Horiguchi. http://git.postgresql.org/pg/commitdiff/f267c1c2447bb8da6e4a6b2fcbb612762c3579a8
  • If the stats collector dies during Hot Standby, restart it. This bug exists as far back as 9.0, when Hot Standby was introduced, so back-patch to all supported branches. Report and patch by Takayuki Tsunakawa, reviewed by Michael Paquier and Kuntal Ghosh. http://git.postgresql.org/pg/commitdiff/4f714b2fd2a580d909607de889ac822956eb8299
  • Fix leftover reference to background writer performing checkpoints. This was changed in PostgreSQL 9.2, but somehow this comment never got updated. http://git.postgresql.org/pg/commitdiff/33839b5ffbd7be56681f31d107ec8238c4a0494a
  • pgstattuple: Don't take heavyweight locks when examining a hash index. It's currently necessary to take a heavyweight lock when scanning a hash bucket, but pgstattuple only examines individual pages, so it doesn't need to do this. If, for some hypothetical reason, it did need to do any heavyweight locking here, this logic would probably still be incorrect, because most of the locks that it is taking are meaningless. Only a heavyweight lock on a primary bucket page has any meaning, but this takes heavyweight locks on all pages regardless of function - and in particular overflow pages, where you might imagine that we'd want to lock the primary bucket page if we needed to lock anything at all. This is arguably a bug that has existed since this code was added in commit dab42382f483c3070bdce14a4d93c5d0cf61e82b, but I'm not going to bother back-patching it because in most cases the only consequence is that running pgstattuple() on a hash index is a little slower than it otherwise might be, which is no big deal. Extracted from a vastly larger patch by Amit Kapila which heavyweight locking for hash indexes entirely; analysis of why this can be done independently of the rest by me. http://git.postgresql.org/pg/commitdiff/d4b5d4caddb73f954d9ef86641201dc99677719d

Bruce Momjian pushed:

Peter Eisentraut pushed:

Heikki Linnakangas pushed:

  • Support multi-dimensional arrays in PL/python. Multi-dimensional arrays can now be used as arguments to a PL/python function (used to throw an error), and they can be returned as nested Python lists. This makes a backwards-incompatible change to the handling of composite types in arrays. Previously, you could return an array of composite types as "[[col1, col2], [col1, col2]]", but now that is interpreted as a two- dimensional array. Composite types in arrays must now be returned as Python tuples, not lists, to resolve the ambiguity. I.e. "[(col1, col2), (col1, col2)]". To avoid breaking backwards-compatibility, when not necessary, () is still accepted for arrays at the top-level, but it is always treated as a single-dimensional array. Likewise, [] is still accepted for composite types, when they are not in an array. Update the documentation to recommend using [] for arrays, and () for composite types, with a mention that those other things are also accepted in some contexts. This needs to be mentioned in the release notes. Alexey Grishchenko, Dave Cramer and me. Reviewed by Pavel Stehule. Discussion: <CAH38_tmbqwaUyKs9yagyRra=SMaT45FPBxk1pmTYcM0TyXGG7Q@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/94aceed317730953476bec490ce0148b2af3c383
  • Give a hint, when [] is incorrectly used for a composite type in array. That used to be accepted, so let's try to give a hint to users on why their PL/python functions no longer work. Reviewed by Pavel Stehule. Discussion: <CAH38_tmbqwaUyKs9yagyRra=SMaT45FPBxk1pmTYcM0TyXGG7Q@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/510e1b8ecf2a6f0d91d50f41f6b7fd75242273a0
  • Fix typos in comments. Vinayak Pokale http://git.postgresql.org/pg/commitdiff/56f39009c53e752493ca4478449a1311865dd51a
  • Fix typo in comment. Daniel Gustafsson http://git.postgresql.org/pg/commitdiff/8a2f08fbeaf75f67da122b49f05f96257df3faed
  • Fix regression test to also work with Python 2. Per buildfarm. http://git.postgresql.org/pg/commitdiff/e131ba4fe58123ce5726c1405486913b429c068c
  • Only treat Python Lists as array dimensions. Instead of treating all python sequence types as array dimensions, except for tuples and various kinds of strings, only treat Python lists as dimensions. The PyBytes_Check() function used previously is only available on Python 2.6 and newer, and it was a bit fiddly anyway. The list of exceptions would require adjustment if Python got a new kind of a sequence similar to bytes/unicodes/strings, so only checking for Lists seems more future-proof. The documentation only mentioned using Lists, so this is closer to what was documented, anyway. This should fix the buildfarm failures on systems building with Python 2.5, although I don't have Python 2.5 installed myself to test with. http://git.postgresql.org/pg/commitdiff/cfd9c87a54b0d414a59a912710c6633313174e51
  • Avoid using platform-dependent floats in test case. The number of decimals printed for floats varied in this test case, as noted by several buildfarm members. There's nothing special about floats and arrays in the code being tested, so replace the floats with numerics to make the output platform-independent. http://git.postgresql.org/pg/commitdiff/73c8e8506cd1933ccf5c5d61088ca171a5f414c0
  • Remove platform-dependent PL/python test case. Turns out that the output format of Python Decimal isn't totally platform- independent either. There are other tests for multi-dimensional arrays, so rather than try to fix this test case, just remove it. Per buildfarm member prairiedog. http://git.postgresql.org/pg/commitdiff/8eb6337f9f53c85e707f60157e42fcacfe927668

Tatsuo Ishii pushed:

Correctifs en attente

Samuel D. Leslie sent in a patch to add a radiustimeout parameter for RADIUS HBA.

Tsutomu Yamada and Ashutosh Bapat traded patches to change a returned NULL to NIL in create_foreignscan_path(), and a similar issue in a create_foreignscan_path() call in add_foreign_grouping_paths().

Petr Jelinek sent in another revision of a patch to implement logical replication.

Petr Jelinek sent in another revision of a patch to follow timeline switches in logical decoding and add a test harness for same.

Laurenz Albe sent in two more revisions of a patch to add PGDLLEXPORT to PG_FUNCTION_INFO_V1 function declarations.

Haribabu Kommi sent in another revision of a patch to add macaddr 64 bit (EUI-64) datatype support.

Robert Haas and Victor Wagner traded patches to implement failover on the libpq connect level.

Peter Geoghegan sent in another revision of a patch to cap the number of tapes used by external sorts, add parallel B-tree index build sorting, and add a force_btree_randomaccess GUC for testing.

Amit Kapila sent in two more revisions of a patch to extend the bufmgr api to include hash index.

Ashutosh Sharma sent in a patch to add microvacuum support for hash indexes.

Amit Kapila sent in another revision of a patch to add group update clog.

Kyotaro HORIGUCHI sent in three more revisions of a patch to implement asynchronous execution.

Noah Misch sent in a patch to firmly recommend PG_GETARG_TEXT_PP() over PG_GETARG_TEXT_P().

Tomas Vondra sent in another revision of a patch to add two slab-like memory allocators.

Venkata B Nagothi sent in a patch to add a recovery_target_incomplete GUC.

Julian Markwort sent in another revision of a patch to allow specifying a password file location for libpq-based applications.

Karl O. Pinc and Gilles Darold traded patches to implement a pg_current_logfile() function.

Heikki Linnakangas and Kyotaro HORIGUCHI traded patches to implement a radix tree for character conversion.

Thomas Munro sent in another revision of a patch to add a replay_lag column to the pg_stat_replication view.

Claudio Freire sent in another revision of a patch to refetch buffers on backward scan and allow using more than 1GB work mem in VACUUM.

Peter Eisentraut sent in a patch to add NLS to the remaining bin programs which were moved from contrib a while ago.

Rushabh Lathia sent in another revision of a patch to implement gather merge.

Etsuro Fujita sent in another revision of a patch to push down more full joins to the postgres FDW.

Ashutosh Bapat sent in another revision of a patch to pgstat to avoid writing on SIGQUIT.

Peter Eisentraut sent in a patch to add a user-callable function to import operation system collations.

Michaël Paquier sent in a patch to apply XLR_INFO_MASK correctly when looking at WAL record information.

Peter Eisentraut sent in a patch to add rules to download raw UNICODE mapping files.

Peter Eisentraut sent in a patch to split up psql's \d modifiers column into more legible pieces.

Haribabu Kommi sent in another revision of a patch to add a pg_hba_file_settings view.

Michaël Paquier sent in another revision of a patch to add a wal_consistency GUC with associated machinery.

Mithun Cy sent in two revisions of a patch to add auto_prewarm.

Peter Eisentraut sent in a patch to save redundant code for pseudotype I/O functions.

Ashutosh Bapat sent in a patch to free unused paths in the case of partition-wise join for join between (declaratively) partitioned tables.

Amit Langote sent in another revision of a patch to implement declarative partitioning for tables.

Julien Rouhaud sent in a patch to remove pq_putmessage_hook and pq_flush_hook.

Magnus Hagander and Michaël Paquier traded patches to fix an issue where symlinks in the wrong place could cause streaming backups to fail.

Tomas Vondra sent in another revision of a patch to implement multivariate statistics.

Michaël Paquier sent in another revision of a patch to mention column name in error messages.

Kevin Grittner sent in another revision of a patch to construct and enable the use of delta relations in AFTER triggers.

Masahiko Sawada sent in another revision of a patch to support 2PC across FDWs.

Kyotaro HORIGUCHI sent in another revision of a patch to refactor tab completion in psql to use a more principled approach to grammar.

par N Bougain le vendredi 4 novembre 2016 à 13h12

mardi 1 novembre 2016

Damien Clochard

PostgreSQL Magazine présente : The Paper Elephant #01

Le projet PostgreSQL Magazine revien avec un concept original nommé project is back The Paper Elephant : un journal pliant et dynamique.

Comme d’habitude, le contenu provient directement de membres de la communauté PostgreSQL : Josh Berkus parle de la nouvelle politique de numérotation, Hans-Jürgen Schönig écrit sur les performances et Craig Kierstens présente un tour d’horizon JSON, JSONB et hstore. Vous trouverez également une interview de Paul Ramsey à propos de PostGIS.

Cette édition a été créée grace à l’aide financière de PostgreSQL Europe et de nombreux contributeurs bénévoles ( rédacteurs, éditeurs, relecteurs ). Un grand merci à tous !

Comme son l’indique The Paper Elephant est avant tout un “media papier” destiné à être imprimé et distribué pendant les événements informatiques (conférences, meetup, etc.) pour promouvoir PostgreSQL.

La première édition sera distribuée à PostgreSQL Conference Europe 2016, Paris Open Source Summit 2016 et FOSDEM 2017. Si vous voulez la distribuer à l’occasion d’une rencontre informatique près de chez vous, Contactez nous !

Pour avoir un aperçu du journal, vous pouvez télécharger la version PDF ici : http://pgmag.org/download

The Paper Elephant est un projet ouvert et collectif. Rejoignez-nous via http://www.pgmag.org/contribute

par Damien Clochard le mardi 1 novembre 2016 à 15h17

jeudi 27 octobre 2016

Actualités PostgreSQL.fr

Sortie de PostgreSQL 9.6.1 et Fin de Vie de PostgreSQL 9.1

Paris, le 27 octobre 2017

Le PostgreSQL Global Development Group a publié une mise à jour pour toutes les versions supportées de notre système de base de données, les versions 9.6.1, 9.5.5, 9.4.10, 9.3.15, 9.2.19 et 9.1.24, où la version 9.1.24 est la dernière de la branche 9.1. Cette version corrige un problème d’enregistrement dans les journaux de transaction (WAL) des relations tronquées pouvant conduire à une corruption de données. Elle résout aussi un certain nombre de bugs rapportés sur les trois derniers mois. Les utilisateurs qui sont affectés par cette corruption de données doivent faire la mise à jour immédiatement. Les autres devraient planifier cette mise à jour à la prochaine interruption de service planifiée.

Enregistrement dans les WAL des relations tronquées

Cette mise à jour corrige l’enregistrement dans les journaux de transaction (WAL) des relations tronquées, et assure maintenant que la carte des epaces libres (Free Space Map ou FSM ) est elle aussi tronquée lorsqu’une commande TRUNCATE est envoyée, qui conduisait à la corruption de données. Si la FSM n’était pas tronquée, une base PostgreSQL en mode réparation (recovery) pouvait retourner une page qui avait déjà été tronquées et retourner une erreur du type :

ERROR:  could not read block 28991 in file "base/16390/572026": read only 0 of 8192 bytes

Si les checksum sont activés, des échecs de checksum sur la carte de visibilité (Visibility Map ou VM) peuvent aussi survenir.

Ce problème est présent dans les séries 9.3, 9.4, 9.5 et 9.6 des publications de PostgreSQL.

Problèmes avec pg_upgrade sur les machines big-endian

Sur les machines big-endian (ex : plusieurs architectures non-Intel), pg_upgrade pourrait écrire de manière incorrecte les octets de la carte de visibilité conduisant pg_upgrade à l’échec.

Ce problème est présent uniquement dans les versions 9.6.0 de PostgreSQL.

Correctifs de bug et améliorations

En plus de ce qui énoncé ci-dessus, cette mise à jour corrige aussi un certain nombre de bugs rapportés sur les derniers mois. Certains problèmes ne touchent que la version 9.6, mais certains touchent toutes les versions supportées. Il y a plus de 50 correctifs dans cette version, incluant :

  • Correction d’un risque d’utilisation de variable après libération (“use-after-free”) dans l’exécution des fonctions d’agrégats utilisant DISTINCT, pouvant conduire à un crash.
  • Correction d’une manipulation incorrecte des agrégats polymorphes utilisés comme fonctions fenêtrées, pouvant conduire à des crashs.
  • Correction de la création incorrecte des index GIN dans les enregistrements WAL sur les machines big-endian
  • Correction de la perte de descripteur de fichier lorsqu’une relation temporaire de plus d’1 Go est tronquée
  • Correction d’une fuite sur la durée de vie d’une requête en mémoire lors d’un UPDATE de masse sur une table avec une PRIMARY KEY ou un index REPLICA IDENTITY
  • Correction des SELECT FOR UPDATE/SHARE pour verrouiller correctement les enregistrements qui ont été mis à jour par une transaction qui a été annulée par la suite
  • Correction de COPY sur la liste de nom de colonne d’une table qui a la sécurité au niveau ligne (RLS) activée
  • Correction de la suppression d’enregistrements TOAST spéculativement insérés en retour d’un INSERT … ON CONFLICT
  • Correction de la longueur du timeout quand un VACUUM est en attente d’un verrou exclusif sur une table pour qu’il puisse tronquer la table
  • Correction de problèmes dans la fusion de contraintes CHECK héritées lorsque une table est créée ou altérée
  • Correction du remplacement d’un tableau d’éléments dans jsonb_set()
  • Correction d’une possible erreur de tri lors de l’interruption de l’utilisation des clés abrégées dans les index btree
  • Sur Windows, nouvel essai de création du segment de control en mémoire partagée dynamique après une erreur sur accès refusé
  • Correction d’une erreur de calcul de la latence moyenne dans pgbench
  • Fait en sorte que pg_receivexlog fonctionne correctement avec –synchronous sans slots
  • Force pg_rewind à désactiver synchronous_commit dans sa session sur le serveur source
  • Suppression des tentatives de partage des contextes SSL au travers de multiples connections dans la libpq
  • Support de OpenSSL 1.1.0
  • Installation de l’infrastructure de tests TAP de façon à ce qu’elle soit disponible pour le test des extensions
  • Plusieurs corrections pour le décodage logique des WAL et les slots de réplication
  • Plusieurs corrections de problèmes mineurs dans pg_dump, pg_xlogdump, et pg_upgrade
  • Plusieurs corrections de problèmes mineurs dans le planificateur de requêtes et dans la sortie d’EXPLAIN
  • Plusieurs corrections dans le support de timezone

Cette mise à jour contient aussi le tzdata 2016h pour les changements légaux de DST en Palestine et Turquie, et des corrections historiques pour la Turquie et quelques régions de Russie. Basculement sur des abbréviations numériques pour quelques time zones en Antarctica, l’ancienne Union Soviétique, et le Sri Lanka.

La base de données des time zones de l’IANA renvoyait précédemment des abréviations textuelles pour les time zones, faisant que parfois ces abréviations n’avaient que peu ou pas cours auprès de la population locale. L’IANA est en cours de renversement de cette politique en faveur de l’utilisation des offsets UTC numériques dans les zones où il n’y a pas d’évidence de l’utilisation d’abréviation anglaise dans le monde réel. Pour le moment tout au moins, PostgreSQL continuera d’accepter les abréviations supprimées pour l’entrée de timestamp. Mais elles ne seront pas montrées dans la vue pg_timezone_names ni utilisés en sortie.

Dans cette mise à jour, AMT n’est plus montrée comme étant en cours d’utilisation pour signifier le temps en Arménie. Par conséquent, nous avons changé l’abréviation pour quelle soit interprétée par défaut comme Amazon Time, donc UTC-4 et pas UTC+4.

Notification d’EOL (End Of Life, fin de vie) pour la version 9.1.

PostgreSQL 9.1 est maintenant en fin de vie (EOL). Le projet prévoit de ne plus sortir aucune version supplémentaire pour cette version. Nous encourageons fortement les utilisateurs à commencer à planifier une mise à jour vers une version plus récente dès que possible. Voyez la politique de version pour plus d’informations

Mise à jour

Toutes les mises à jour de PostgreSQL sont cumulatives. Comme pour les autres versions mineures, les utilisateurs n’ont pas besoin d’exporter et réimporter leur base, ni d’utiliser pg_upgrade pour appliquer cette mise à jour. Vous pouvez simplement éteindre PostgreSQL et mettre à jour les binaires.

Si votre système était affecté par le bug pg_upgrade sur big-endian, merci de lire https://wiki.postgresql.org/wiki/Visibility_Map_Problems et suivez les instructions sur comment corriger ce problème sur vos instances PostgreSQL.

Les utilisateurs qui ont sauté une ou plusieurs mises à jour mineures pourraient avoir besoin de lancer des étapes supplémentaires après la mise à jour; merci de consulter les notes de version des versions précédentes pour plus de détails.

Liens

par daamien le jeudi 27 octobre 2016 à 13h14

mercredi 26 octobre 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 23 octobre 2016

PostgreSQL@SCaLE est un événement de 2 jours à double programmes qui aura lieu les 2 & 3 mars 2017 au centre de convention de Pasadena, intégré au SCaLE 15X. L'appel à conférenciers court jusqu'au 15 novembre 2016 : http://www.socallinuxexpo.org/scale/15x/cfp

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en octobre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161023230118.GA30097@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

  • Replace PostmasterRandom() with a stronger way of generating randomness. This adds a new routine, pg_strong_random() for generating random bytes, for use in both frontend and backend. At the moment, it's only used in the backend, but the upcoming SCRAM authentication patches need strong random numbers in libpq as well. pg_strong_random() is based on, and replaces, the existing implementation in pgcrypto. It can acquire strong random numbers from a number of sources, depending on what's available: - OpenSSL RAND_bytes(), if built with OpenSSL - On Windows, the native cryptographic functions are used - /dev/urandom - /dev/random Original patch by Magnus Hagander, with further work by Michael Paquier and me. Discussion: <CAB7nPqRy3krN8quR9XujMVVHYtXJ0_60nqgVc6oUk8ygyVkZsA@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/9e083fd4683294f41544e6d0d72f6e258ff3a77c
  • Fix use-after-free around DISTINCT transition function calls. Have tuplesort_gettupleslot() copy the contents of its current table slot as needed. This is based on an approach taken by tuplestore_gettupleslot(). In the future, tuplesort_gettupleslot() may also be taught to avoid copying the tuple where caller can determine that that is safe (the tuplestore_gettupleslot() interface already offers this option to callers). Patch by Peter Geoghegan. Fixes bug #14344, reported by Regina Obe. Report: <20160929035538.20224.39628@wrigleys.postgresql.org> Backpatch-through: 9.6 http://git.postgresql.org/pg/commitdiff/d8589946ddd5c43e1ebd01c5e92d0e177cbfc198
  • Revert "Replace PostmasterRandom() with a stronger way of generating randomness." This reverts commit 9e083fd4683294f41544e6d0d72f6e258ff3a77c. That was a few bricks shy of a load: * Query cancel stopped working * Buildfarm member pademelon stopped working, because the box doesn't have /dev/urandom nor /dev/random. This clearly needs some more discussion, and a quite different patch, so revert for now. http://git.postgresql.org/pg/commitdiff/faae1c918e8aaae034eaf3ea103fcb6ba9adc5ab
  • Fix WAL-logging of FSM and VM truncation. When a relation is truncated, it is important that the FSM is truncated as well. Otherwise, after recovery, the FSM can return a page that has been truncated away, leading to errors like: ERROR: could not read block 28991 in file "base/16390/572026": read only 0 of 8192 bytes We were using MarkBufferDirtyHint() to dirty the buffer holding the last remaining page of the FSM, but during recovery, that might in fact not dirty the page, and the FSM update might be lost. To fix, use the stronger MarkBufferDirty() function. MarkBufferDirty() requires us to do WAL-logging ourselves, to protect from a torn page, if checksumming is enabled. Also fix an oversight in visibilitymap_truncate: it also needs to WAL-log when checksumming is enabled. Analysis by Pavan Deolasee. Discussion: <CABOikdNr5vKucqyZH9s1Mh0XebLs_jRhKv6eJfNnD2wxTn=_9A@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/917dc7d2393ce680dea7a59418be9ff341df3c14
  • Use OpenSSL EVP API for symmetric encryption in pgcrypto. The old "low-level" API is deprecated, and doesn't support hardware acceleration. And this makes the code simpler, too. Discussion: <561274F1.1030000@iki.fi> http://git.postgresql.org/pg/commitdiff/5ff4a67f63fd6d3eb01ff9707d4674ed54a89f3b

Robert Haas pushed:

  • By default, set log_line_prefix = '%m [%p] '. This value might not be to everyone's taste; in particular, some people might prefer %t to %m, and others may want %u, %d, or other fields. However, it's a vast improvement on the old default of ''. Christoph Berg http://git.postgresql.org/pg/commitdiff/7d3235ba42f8d5fc70c58e242702cc5e2e3549a6
  • Fix typo in comment. Amit Langote http://git.postgresql.org/pg/commitdiff/fca41acb86902b90218dcc3bc0ffc462850a090f
  • Improve regression test coverage for hash indexes. On my system, this improves coverage for src/backend/access/hash from 61.3% of lines to 88.2% of lines, and from 83.5% of functions to 97.5% of functions, which is pretty good for 36 lines of tests. Mithun Cy, reviewing by Amit Kapila and Álvaro Herrera http://git.postgresql.org/pg/commitdiff/b801e120080de836b834c1b756c4c4d81ce841b5
  • Remove a comment which is now incorrect. Before 5d305d86bd917723f09ab4f15c075d90586a210a, this comment was correct, but now it says we do something which we don't actually do. Accordingly, remove the comment. http://git.postgresql.org/pg/commitdiff/ec7db2b483e0ff247ed41612cdb5716022401fe6
  • ename "pg_xlog" directory to "pg_wal". "xlog" is not a particularly clear abbreviation for "write-ahead log", and it sometimes confuses users into believe that the contents of the "pg_xlog" directory are not critical data, leading to unpleasant consequences. So, rename the directory to "pg_wal". This patch modifies pg_upgrade and pg_basebackup to understand both the old and new directory layouts; the former is necessary given the purpose of the tool, while the latter merely avoids an unnecessary backward-compatibility break. We may wish to consider renaming other programs, switches, and functions which still use the old "xlog" naming to also refer to "wal". However, that's still under discussion, so let's do just this much for now. Discussion: CAB7nPqTeC-8+zux8_-4ZD46V7YPwooeFxgndfsq5Rg8ibLVm1A@mail.gmail.com Michael Paquier http://git.postgresql.org/pg/commitdiff/f82ec32ac30ae7e3ec7c84067192535b2ff8ec0e
  • postgres_fdw: Push down aggregates to remote servers. Now that the upper planner uses paths, and now that we have proper hooks to inject paths into the upper planning process, it's possible for foreign data wrappers to arrange to push aggregates to the remote side instead of fetching all of the rows and aggregating them locally. This figures to be a massive win for performance, so teach postgres_fdw to do it. Jeevan Chalke and Ashutosh Bapat. Reviewed by Ashutosh Bapat with additional testing by Prabhat Sahu. Various mostly cosmetic changes by me. http://git.postgresql.org/pg/commitdiff/7012b132d07c2b4ea15b0b3cb1ea9f3278801d98
  • Fix comment formatting. http://git.postgresql.org/pg/commitdiff/919c811ca1e2a545cb1db243af93d55270d84469
  • postgres_fdw: Attempt to stabilize regression results. Set enable_hashagg to false for tests involving least_agg(), so that we get the same plan regardless of local costing variances. Also, remove a test involving sqrt(); it's there to test deparsing of HAVING clauses containing expressions, but that's tested elsewhere anyway, and sqrt(2) deparses with different amounts of precision on different machines. Per buildfarm. http://git.postgresql.org/pg/commitdiff/ad13a09d762f0c903a52ed0ec668a0ba51a61047

Tom Lane pushed:

  • Fix cidin() to handle values above 2^31 platform-independently. CommandId is declared as uint32, and values up to 4G are indeed legal. cidout() handles them properly by treating the value as unsigned int. But cidin() was just using atoi(), which has platform-dependent behavior for values outside the range of signed int, as reported by Bart Lengkeek in bug #14379. Use strtoul() instead, as xidin() does. In passing, make some purely cosmetic changes to make xidin/xidout look more like cidin/cidout; the former didn't have a monopoly on best practice IMO. Neither xidin nor cidin make any attempt to throw error for invalid input. I didn't change that here, and am not sure it's worth worrying about since neither is really a user-facing type. The point is just to ensure that indubitably-valid inputs work as expected. It's been like this for a long time, so back-patch to all supported branches. Report: <20161018152550.1413.6439@wrigleys.postgresql.org> http://git.postgresql.org/pg/commitdiff/6f13a682c86801cfb9ae4f3126888b42f3cb5c46
  • Update time zone data files to tzdata release 2016g. DST law changes in Turkey. Historical corrections for America/Los_Angeles, Europe/Kirov, Europe/Moscow, Europe/Samara, and Europe/Ulyanovsk. Rename Asia/Rangoon to Asia/Yangon, with a backward compatibility link. The IANA crew continue their campaign to replace invented time zone abbrevations with numeric GMT offsets. This update changes numerous zones in Antarctica and the former Soviet Union, for instance Antarctica/Casey now reports "+08" not "AWST" in the pg_timezone_names view. I kept these abbreviations in the tznames/ data files, however, so that we will still accept them for input. (We may want to start trimming those files someday, but today is not that day.) An exception is that since IANA no longer claims that "AMT" is in use in Armenia for GMT+4, I replaced it in the Default file with GMT-4, corresponding to Amazon Time which is in use in South America. It may be that that meaning is also invented and IANA will drop it in a future update; but for now, it seems silly to give pride of place to a meaning not traceable to IANA over one that is. http://git.postgresql.org/pg/commitdiff/ecbac3e6e038e990f24a2e0eacdcd6738292105f
  • Suppress "Factory" zone in pg_timezone_names view for tzdata >= 2016g. IANA got rid of the really silly "abbreviation" and replaced it with one that's only moderately silly. But it's still pointless, so keep on not showing it. http://git.postgresql.org/pg/commitdiff/a3215431ab7c667bf581728f10c80a36abbe1d5a
  • Windows portability fix. Per buildfarm. http://git.postgresql.org/pg/commitdiff/ad90ac4d671d320ade3c127f215e97cd49c307fb
  • Sync our copy of the timezone library with IANA release tzcode2016g. This is mostly to absorb some corner-case fixes in zic for year-2037 timestamps. The other changes that have been made are unlikely to affect our usage, but nonetheless we may as well take 'em. http://git.postgresql.org/pg/commitdiff/f3094920a567cde6c86adf36a1a033d7431b11ff
  • Another portability fix for tzcode2016g update. clang points out that SIZE_MAX wouldn't fit into an int, which means this comparison is pretty useless. Per report from Thomas Munro. http://git.postgresql.org/pg/commitdiff/23ed2ba8121178474f8c51774c6c258cb165a562
  • Update time zone data files to tzdata release 2016h. (Didn't I just do this? Oh well.) DST law changes in Palestine. Historical corrections for Turkey. Switch to numeric abbreviations for Asia/Colombo. http://git.postgresql.org/pg/commitdiff/d8fc45bd0f62fcebac80c63840b753f8e3b737ff
  • Sync our copy of the timezone library with IANA release tzcode2016h. This absorbs a fix for a symlink-manipulation bug in zic that was introduced in 2016g. It probably isn't interesting for our use-case, but I'm not quite sure, so let's update while we're at it. http://git.postgresql.org/pg/commitdiff/5e21b6811148fdc1fce9dcdcdc777418cc901fe4
  • Fix EXPLAIN so that it doesn't emit invalid XML in corner cases. With track_io_timing = on, EXPLAIN (ANALYZE, BUFFERS) will emit fields named like "I/O Read Time". The slash makes that invalid as an XML element name, so that adding FORMAT XML would produce invalid XML. We already have code in there to translate spaces to dashes, so let's generalize that to convert anything that isn't a valid XML name character, viz letters, digits, hyphens, underscores, and periods. We could just reject slashes, which would run a bit faster. But the fact that this went unnoticed for so long doesn't give me a warm feeling that we'd notice the next creative violation, so let's make it a permanent fix. Reported by Markus Winand, though this isn't his initial patch proposal. Back-patch to 9.2 where track_io_timing was added. The problem is only latent in 9.1, so I don't feel a need to fix it there. Discussion: <E0BF6A45-68E8-45E6-918F-741FB332C6BB@winand.at> http://git.postgresql.org/pg/commitdiff/709e461befa8a4999c4ccdbfc7260ef8092e805c
  • Doc: wording tweak for PERL, PYTHON, TCLSH configuration variables. Replace "Full path to ..." with "Full path name of ...". At least one user has misinterpreted the existing wording as meaning "Directory containing ...". http://git.postgresql.org/pg/commitdiff/7aa2c10ac6785a2de683609b98da607e588a6d02
  • First-draft release notes for 9.6.1. As usual, the release notes for other branches will be made by cutting these down, but put them up for community review first. http://git.postgresql.org/pg/commitdiff/eacaf6e29fd2a3047aff9738a35a8e9b05e55375
  • Improve documentation about use of Linux huge pages. Show how to get the system's huge page size, rather than misleadingly referring to PAGE_SIZE (which is usually understood to be the regular page size). Show how to confirm whether huge pages have been allocated. Minor wordsmithing. Back-patch to 9.4 where this section appeared. http://git.postgresql.org/pg/commitdiff/1885c88459698251eca64f095d9942c540ba0fa8
  • Avoid testing tuple visibility without buffer lock in RI_FKey_check(). Despite the argumentation I wrote in commit 7a2fe85b0, it's unsafe to do this, because in corner cases it's possible for HeapTupleSatisfiesSelf to try to set hint bits on the target tuple; and at least since 8.2 we have required the buffer content lock to be held while setting hint bits. The added regression test exercises one such corner case. Unpatched, it causes an assertion failure in assert-enabled builds, or otherwise would cause a hint bit change in a buffer we don't hold lock on, which given the right race condition could result in checksum failures or other data consistency problems. The odds of a problem in the field are probably pretty small, but nonetheless back-patch to all supported branches. Report: <19391.1477244876@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/6292c2339186bac215bab5a1f01370f9735582c1

Andres Freund pushed:

Peter Eisentraut pushed:

Magnus Hagander pushed:

Correctifs en attente

Haribabu Kommi and Vinayak Pokale traded patches to add a pg_stat_sql system view.

Dilip Kumar sent in two more revisions of a patch to add parallel bitmap heap scan.

Masahiko Sawada sent in another revision of a patch to add quorum commit for multiple synchronous replication.

Kyotaro HORIGUCHI sent in another revision of a patch to implement asynchronous execution.

Laurenz Albe sent in a patch to add PGDLLEXPORT to a sample C function.

Ashutosh Bapat and Etsuro Fujita traded patches to ensure that altering a foreign table invalidates plans involving same.

Aleksander Alekseev sent in two revisions of a patch to enable logging the contents of COPY statements.

Heikki Linnakangas and Michaël Paquier traded patches to fix an FSM corruption leading to errors.

Gilles Darold sent in another revision of a patch to implement pg_current_logfile().

Dmitry Dolgov sent in another revision of a patch to implement generic type subscripting.

Michaël Paquier sent in two more revisions of a patch to implement SCRAM auth.

Rushabh Lathia sent in another revision of a patch to add Gather Merge.

Thom Brown sent in another revision of a patch to implement failover on the libpq connect level.

Jeevan Chalke and Ashutosh Bapat traded patches to implement aggregate pushdown.

Peter Geoghegan sent in a patch to fix ON CONFLICT bugs at higher isolation levels.

Oleksandr Shulgin sent in a patch to prevent psql from sending commands after a connection reset.

Vinayak Pokale sent in a patch to fix a typo in pgstat.h.

Vinayak Pokale sent in a patch to fix a typo in pgstat.c.

Constantin S. Pan and Michaël Paquier traded patches to fix the fact that there can be a lot of orphan temp tables.

Tomas Vondra sent in two more revisions of a patch to add two slab-like memory allocators.

Kyotaro HORIGUCHI sent in another revision of a patch to implement a radix tree for character conversion.

Masahiko Sawada sent in a patch to allow specifying the log file name of pgbench -l option.

Julien Rouhaud sent in a patch to fix the fact that when track_commit_timestamp is enabled, the oldestCommitTsXid and newestCommitTsXid don't persist after a server restart.

Bruce Momjian sent in a patch to mention pg_reload_conf() as a way to reload configurations in the docs.

Ashutosh Bapat sent in another revision of a patch to allow pushing down more FULL JOINs to the the PostgreSQL FDW.

Michaël Paquier sent in another revision of a patch to rename pg_clog to pg_xact and pg_subtrans to pg_subxact.

par N Bougain le mercredi 26 octobre 2016 à 21h18

vendredi 21 octobre 2016

Nicolas Gollet

PG Session #8 les videos à télécharger

Les vidéos de la "PostgreSQL Session #8" organisée par Dalibo et Oslandia viennent d'être mise en ligne sur youtube !

Afin de vous permettre de visionner ces vidéos partout (dans le train, l'avions, aux fins font de l'Ardèche ;) ), je vous propose de pouvoir les télécharger au format "mp4" directement depuis ce partage one drive.

Au programme 8 conférences :
1. Paul Leroux - Postgres comme NoSql Db - le meilleur des deux mondes
2. Nicolas Ribot - Traitements spatiaux parallélisés pour les gros volumes de données
3. Ludovic Delauné - Point Clouds - modèles, chargement, stockage et manipulation de données
4. Tom van Tilburg - From point clouds and vectordata to 3D models with the help of postgres
5. Giuseppe Broccolo & Julien Rouhaud - BRIN indexes on geospatial database
6. Giulio Calacoci - Step up and face the disaster (pgbarman)
7. Julien Tachoires - Temboard - supervision, administration, dashboard et bien plus
8. Olivier Courtin - De l'analyse spatiale avancée à la GéoDataScience en environnement PostgreSQL - PostGIS

Ces vidéos ont été produites par Dalibo et sont sous licence Creative Commons BY-NC-SA n'hésitez donc pas à les partager!

Merci à Télé Millevaches pour leur travail de captation vidéo et de montage ;)

Bon visionnage à toutes et à tous!

par Nicolas GOLLET le vendredi 21 octobre 2016 à 10h34

mardi 18 octobre 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 16 octobre 2016

[ndt: Meetup à Paris le 10 novembre : http://www.meetup.com/fr-FR/PostgreSQL-User-Group-Paris/]

Offres d'emplois autour de PostgreSQL en octobre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161016222644.GB13559@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Peter Eisentraut pushed:

Heikki Linnakangas pushed:

Tom Lane pushed:

  • In PQsendQueryStart(), avoid leaking any left-over async result. Ordinarily there would not be an async result sitting around at this point, but it appears that in corner cases there can be. Considering all the work we're about to launch, it's hardly going to cost anything noticeable to check. It's been like this forever, so back-patch to all supported branches. Report: <CAD-Qf1eLUtBOTPXyFQGW-4eEsop31tVVdZPu4kL9pbQ6tJPO8g@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/886f6c5ccdb500eeeec7e0abdf1500e20a304c45
  • Update user docs for switch to POSIX semaphores. Since commit ecb0d20a9 hasn't crashed and burned, here's the promised docs update for it. In addition to explaining that Linux and FreeBSD ports now use POSIX semaphores, I did some wordsmithing on pre-existing wording; in particular trying to clarify which SysV parameters need to be set with an eye to total usage across all applications. http://git.postgresql.org/pg/commitdiff/3d21f08bccd316c3850a1943c1ee1e381dab1588
  • Improve documentation for CREATE RECURSIVE VIEW. It was perhaps not entirely clear that internal self-references shouldn't be schema-qualified even if the view name is written with a schema. Spell it out. Discussion: <871sznz69m.fsf@metapensiero.it> http://git.postgresql.org/pg/commitdiff/e34318725ca5b274efd6f57ea7460e89f4dca9f9
  • Docs: grammatical fix. Fix poor grammar introduced in 741ccd501. http://git.postgresql.org/pg/commitdiff/c7e56811fa38cbc39efd6bdd4bb45f2f0444803e
  • Remove "sco" and "unixware" ports. SCO OpenServer and SCO UnixWare are more or less dead platforms. We have never had a buildfarm member testing the "sco" port, and the last "unixware" member was last heard from in 2012, so it's fair to doubt that the code even compiles anymore on either one. Remove both ports. We can always undo this if someone shows up with an interest in maintaining and testing these platforms. Discussion: <17177.1476136994@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/2b860f52ed1b1784cdf3f03886805f5bf250ea74
  • Drop server support for FE/BE protocol version 1.0. While this isn't a lot of code, it's been essentially untestable for a very long time, because libpq doesn't support anything older than protocol 2.0, and has not since release 6.3. There's no reason to believe any other client-side code still uses that protocol, either. Discussion: <2661.1475849167@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/2f1eaf87e868a1c42f2b159958623daa6a666de4
  • Provide DLLEXPORT markers for C functions via PG_FUNCTION_INFO_V1 macro. This isn't really necessary for our own code, because we use a .DEF file in MSVC builds (see gendef.pl), or --export-all-symbols in MinGW and Cygwin builds, to ensure that all global symbols in loadable modules will be exported on Windows. However, third-party authors might use different build processes that need this marker, and it's harmless enough for our own builds. To some extent, this is an oversight in commit e7128e8db, so back-patch to 9.4 where that was added. Laurenz Albe. Discussion: <A737B7A37273E048B164557ADEF4A58B539300BD@ntex2010a.host.magwien.gv.at> http://git.postgresql.org/pg/commitdiff/8518583cdb10340bab3464121629a1a9ec387afa
  • Remove unnecessary int2vector-specific hash function and equality operator. These functions were originally added in commit d8cedf67a to support use of int2vector columns as catcache lookup keys. However, there are no catcaches that use such columns. (Indeed I now think it must always have been dead code: a catcache with such a key column would need an underlying unique index on the column, but we've never had an int2vector btree opclass.) Getting rid of the int2vector-specific operator and function does not lose any functionality, because operations on int2vectors will now fall back to the generic anyarray support. This avoids a wart that a btree index on an int2vector column (made using anyarray_ops) would fail to match equality searches, because int2vectoreq wasn't a member of the opclass. We don't really care much about that, since int2vector is not meant as a type for users to use, but it's silly to have extra code and less functionality. If we ever do want a catcache to be indexed by an int2vector column, we'd need to put back full btree and hash opclasses for int2vector, comparable to the support for oidvector. (The anyarray code can't be used at such a low level, because it needs to do catcache lookups.) But we'll deal with that if/when the need arises. Also worth noting is that removal of the hash int2vector_ops opclass will break any user-created hash indexes on int2vector columns. While hash anyarray_ops would serve the same purpose, it would probably not compute the same hash values and thus wouldn't be on-disk-compatible. Given that int2vector isn't a user-facing type and we're planning other incompatible changes in hash indexes for v10 anyway, this doesn't seem like something to worry about, but it's probably worth mentioning here. Amit Langote Discussion: <d9bb74f8-b194-7307-9ebd-90645d377e45@lab.ntt.co.jp> http://git.postgresql.org/pg/commitdiff/5c80642aa8de8393b08cd3cbf612b325cedd98dc
  • Remove pg_dump/pg_dumpall support for dumping from pre-8.0 servers. The need for dumping from such ancient servers has decreased to about nil in the field, so let's remove all the code that catered to it. Aside from removing a lot of boilerplate variant queries, this allows us to not have to cope with servers that don't have (a) schemas or (b) pg_depend. That means we can get rid of assorted squishy code around that. There may be some nonobvious additional simplifications possible, but this patch already removes about 1500 lines of code. I did not remove the ability for pg_restore to read custom-format archives generated by these old versions (and light testing says that that does still work). If you have an old server, you probably also have a pg_dump that will work with it; but you have an old custom-format backup file, that might be all you have. It'd be possible at this point to remove fmtQualifiedId()'s version argument, but I refrained since that would affect code outside pg_dump. Discussion: <2661.1475849167@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/64f3524e2c8deebc02808aa5ebdfa17859473add
  • pg_dump's getTypes() needn't retrieve typinput or typoutput anymore. Commit 64f3524e2 removed the stanza of code that examined these values. I failed to notice they were unnecessary because my compiler didn't warn about the un-read variables. Noted by Peter Eisentraut. http://git.postgresql.org/pg/commitdiff/c0a3b211bcb790a8d76022cb2b3ffe9795aaf5e9
  • Revert addition of PGDLLEXPORT in PG_FUNCTION_INFO_V1 macro. This turns out not to be as harmless as I thought: MSVC will complain if it sees an "extern" declaration without PGDLLEXPORT and then one with. (Seems fairly silly, given that this can be changed after the fact by the linker, but there you have it.) Therefore, contrib modules that have extern's for V1 functions in header files are falling over in the buildfarm, since none of those externs are marked PGDLLEXPORT. We might or might not conclude that we're willing to plaster those declarations with PGDLLEXPORT in HEAD, but in any case there's no way we're going to ship this change in the back branches. Third-party authors would not thank us for breaking their code in a minor release. Hence, revert the addition of PGDLLEXPORT (but let's keep the extra info in the comment). If we do the other changes we can revert this commit in HEAD. Per buildfarm. http://git.postgresql.org/pg/commitdiff/4f52fd3c6d28a4380a5afc51ae6f774c91837a38
  • Fix broken jsonb_set() logic for replacing array elements. Commit 0b62fd036 did a fairly sloppy job of refactoring setPath() to support jsonb_insert() along with jsonb_set(). In its defense, though, there was no regression test case exercising the case of replacing an existing element in a jsonb array. Per bug #14366 from Peng Sun. Back-patch to 9.6 where bug was introduced. Report: <20161012065349.1412.47858@wrigleys.postgresql.org> http://git.postgresql.org/pg/commitdiff/9c4cc9e2c75e84a8267396be5cccbfe25b8f63f6
  • Fix pg_dumpall regression test to be locale-independent. The expected results in commit b4fc64578 seem to have been generated in a non-C locale, which just points up the fact that the ORDER BY clause was locale-sensitive. Per buildfarm. http://git.postgresql.org/pg/commitdiff/0a4bf6b192377bef8e69c92d0c95434a91509f12
  • Clean up handling of anonymous mmap'd shared-memory segment. Fix detaching of the mmap'd segment to have its own on_shmem_exit callback, rather than piggybacking on the one for detaching from the SysV segment. That was confusing, and given the distance between the two attach calls, it was trouble waiting to happen. Make the detaching calls idempotent by clearing AnonymousShmem to show we've already unmapped. I spent quite a bit of time yesterday trying to find a path that would allow the munmap()'s to be done twice, and while I did not succeed, it seems silly that there's even a question. Make the #ifdef logic less confusing by separating "do we want to use anonymous shmem" from EXEC_BACKEND. Even though there's no current scenario where those conditions are different, it is not helpful for different places in the same file to be testing EXEC_BACKEND for what are fundamentally different reasons. Don't do on_exit_reset() in StartBackgroundWorker(). At best that's useless (InitPostmasterChild would have done it already) and at worst it could zap some callback that's unrelated to shared memory. Improve comments, and simplify the huge_pages enablement logic slightly. Back-patch to 9.4 where hugepage support was introduced. Arguably this should go into 9.3 as well, but the code looks significantly different there, and I doubt it's worth the trouble of adapting the patch given I can't show a live bug. http://git.postgresql.org/pg/commitdiff/15fc5e15811337f5a81d4ae44c6149256f6dd15f
  • Try to find out the actual hugepage size when making a MAP_HUGETLB request. Even if Linux's mmap() is okay with a partial-hugepage request, munmap() is not, as reported by Chris Richards. Therefore it behooves us to try a bit harder to find out the actual hugepage size, instead of assuming that we can skate by with a guess. For the moment, just look into /proc/meminfo to find out the default hugepage size, and use that. Later, on kernels that support requests for nondefault sizes, we might try to consider other alternatives. But that smells more like a new feature than a bug fix, especially if we want to provide any way for the DBA to control it, so leave it for another day. I set this up to allow easy addition of platform-specific code for non-Linux platforms, if needed; but right now there are no reports suggesting that we need to work harder on other platforms. Back-patch to 9.4 where hugepage support was introduced. Discussion: <31056.1476303954@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/cb775768e3e37d466d69b7177a92508b81c1c204
  • Remove dead code in pg_dump. I'm not sure if this provision for "pg_backup" behaving a bit differently from "pg_dump" ever did anything useful in a released version. But it's definitely dead code now. Michael Paquier http://git.postgresql.org/pg/commitdiff/c08521eb55135c493cee23541c233870cdff14b7
  • Fix another bug in merging of inherited CHECK constraints. It's not good for an inherited child constraint to be marked connoinherit; that would result in the constraint not propagating to grandchild tables, if any are created later. The code mostly prevented this from happening but there was one case that was missed. This is somewhat related to commit e55a946a8, which also tightened checks on constraint merging. Hence, back-patch to 9.2 like that one. This isn't so much because there's a concrete feature-related reason to stop there, as to avoid having more distinct behaviors than we have to in this area. Amit Langote Discussion: <b28ee774-7009-313d-dd55-5bdd81242c41@lab.ntt.co.jp> http://git.postgresql.org/pg/commitdiff/3cca13cbfcc5312f7ae1728213e197c6f37ca62a
  • Fix handling of pgstat counters for TRUNCATE in a prepared transaction. pgstat_twophase_postcommit is supposed to duplicate the math in AtEOXact_PgStat, but it had missed out the bit about clearing t_delta_live_tuples/t_delta_dead_tuples for a TRUNCATE. It's harder than you might think to replicate the issue here, because those counters would only be nonzero when a previous transaction in the same backend had added/deleted tuples in the truncated table, and those counts hadn't been sent to the stats collector yet. Evident oversight in commit d42358efb. I've not added a regression test for this; we tried to add one in d42358efb, and had to revert it because it was too timing-sensitive for the buildfarm. Back-patch to 9.5 where d42358efb came in. Stas Kelvich Discussion: <EB57BF68-C06D-4737-BDDC-4BA778F4E62B@postgrespro.ru> http://git.postgresql.org/pg/commitdiff/81e82a2bd48865f9a294b63d9492f9fde6a32787
  • Fix assorted integer-overflow hazards in varbit.c. bitshiftright() and bitshiftleft() would recursively call each other infinitely if the user passed INT_MIN for the shift amount, due to integer overflow in negating the shift amount. To fix, clamp to -VARBITMAXLEN. That doesn't change the results since any shift distance larger than the input bit string's length produces an all-zeroes result. Also fix some places that seemed inadequately paranoid about input typmods exceeding VARBITMAXLEN. While a typmod accepted by anybit_typmodin() will certainly be much less than that, at least some of these spots are reachable with user-chosen integer values. Andreas Seltenreich and Tom Lane Discussion: <87d1j2zqtz.fsf@credativ.de> http://git.postgresql.org/pg/commitdiff/32fdf42cf546f613aab9ca98935c40a046187fa9

Andres Freund pushed:

  • Make regression tests less dependent on hash table order. Upcoming changes to the hash table code used, among others, for grouping and set operations will change the output order for a few queries. To make it less likely that actual bugs are hidden between regression test ordering changes, and to make the tests robust against platform dependant ordering, add ORDER BYs guaranteeing the output order. As it's possible that some of the changes expose platform dependant ordering, push this earlier, to let the buildfarm shake out potentially unstable results. Discussion: <20160727004333.r3e2k2y6fvk2ntup@alap3.anarazel.de> http://git.postgresql.org/pg/commitdiff/0137caf273f4297c4d36df3a542d7c0c853e75be
  • Fix further hash table order dependent tests. Similar to 0137caf273, this makes contrib and pl tests less dependant on hash-table order. After this commit, at least some order affecting changes to execGrouping.c don't result in regression test changes anymore. http://git.postgresql.org/pg/commitdiff/ccbb852cd6101e93976111676a90f1e5d9268951
  • Make pg_dumpall's database ACL query independent of hash table order. Previously GRANT order on databases was not well defined, due to the use of EXCEPT without an ORDER BY. Add an ORDER BY, adapt test output. I don't, at the moment, see reason to backpatch this. http://git.postgresql.org/pg/commitdiff/b4fc645787cc7c614c0c97fc9fffacf2bdc6a388
  • Use more efficient hashtable for tidbitmap.c to speed up bitmap scans. Use the new simplehash.h to speed up tidbitmap.c uses. For bitmap scan heavy queries speedups of over 100% have been measured. Both lossy and exact scans benefit, but the wins are bigger for mostly exact scans. The conversion is mostly trivial, except that tbm_lossify() now restarts lossifying at the point it previously stopped. Otherwise the hash table becomes unbalanced because the scan in done in hash-order, leaving the end of the hashtable more densely filled then the beginning. That caused performance issues with dynahash as well, but due to the open chaining they were less pronounced than with the linear adressing from simplehash.h. Reviewed-By: Tomas Vondra Discussion: <20160727004333.r3e2k2y6fvk2ntup@alap3.anarazel.de> http://git.postgresql.org/pg/commitdiff/75ae538bc3168bf44475240d4e0487ee2f3bb376
  • Add likely/unlikely() branch hint macros. These are useful for very hot code paths. Because it's easy to guess wrongly about likelihood, and because such likelihoods change over time, they should be used sparingly. Past tests have shown it'd be a good idea to use them in some places, e.g. in error checks around ereports that ERROR out, but that's work for later. Discussion: <20160727004333.r3e2k2y6fvk2ntup@alap3.anarazel.de> http://git.postgresql.org/pg/commitdiff/aa3ca5e3dd60bf0b992b74f955378f28e601292a
  • Add a macro templatized hashtable. dynahash.c hash tables aren't quite fast enough for some use-cases. There are several reasons for lacking performance: The use of chaining for collision handling makes them cache inefficient, that's especially an issue when the tables get bigger. As the element sizes for dynahash are only determined at runtime, offset computations are somewhat expensive Hash and element comparisons are indirect function calls, causing unnecessary pipeline stalls It's two level structure has some benefits (somewhat natural partitioning), but increases the number of indirections to fix several of these the hash tables have to be adjusted to the individual use-case at compile-time. C unfortunately doesn't provide a good way to do compile code generation (like e.g. c++'s templates for all their weaknesses do). Thus the somewhat ugly approach taken here is to allow for code generation using a macro-templatized header file, which generates functions and types based on a prefix and other parameters. Later patches use this infrastructure to use such hash tables for tidbitmap.c (bitmap scans) and execGrouping.c (hash aggregation, ...). In queries where these use up a large fraction of the time, this has been measured to lead to performance improvements of over 100%. There are other cases where this could be useful (e.g. catcache.c). The hash table design chosen is a variant of linear open-addressing. The biggest disadvantage of simple linear addressing schemes are highly variable lookup times due to clustering, and deletions leaving a lot of tombstones around. To address these issues a variant of "robin hood" hashing is employed. Robin hood hashing optimizes chaining lengths by moving elements close to their optimal bucket ("rich" elements), out of the way if a to-be-inserted element is further away from its optimal position (i.e. it's "poor"). While that can make insertions slower, the average lookup performance is a lot better, and higher fill factors can be used in a still performant manner. To avoid tombstones - which normally solve the issue that a deleted node's presence is relevant to determine whether a lookup needs to continue looking or is done - buckets following a deleted element are shifted backwards, unless they're empty or already at their optimal position. There's further possible improvements that can be made to this implementation. Amongst others: Use distance as a termination criteria during searches. This is generally a good idea, but I've been able to see the overhead of distance calculations in some cases. Consider combining the 'empty' status into the hashvalue, and enforce storing the hashvalue. That could, in some cases, increase memory density and remove a few instructions. Experiment further with the, very conservatively choosen, fillfactor. Make maximum size of hashtable configurable, to allow storing very very large tables. That'd require 64bit hash values to be more common than now, though. some smaller memcpy calls could be optimized to copy larger chunks. But since the new implementation is already considerably faster than dynahash it seem sensible to start using it. Reviewed-By: Tomas Vondra Discussion: <20160727004333.r3e2k2y6fvk2ntup@alap3.anarazel.de> http://git.postgresql.org/pg/commitdiff/b30d3ea824c5ccba43d3e942704f20686e7dbab8
  • Use more efficient hashtable for execGrouping.c to speed up hash aggregation. The more efficient hashtable speeds up hash-aggregations with more than a few hundred groups significantly. Improvements of over 120% have been measured. Due to the the different hash table queries that not fully determined (e.g. GROUP BY without ORDER BY) may change their result order. The conversion is largely straight-forward, except that, due to the static element types of simplehash.h type hashes, the additional data some users store in elements (e.g. the per-group working data for hash aggregaters) is now stored in TupleHashEntryData->additional. The meaning of BuildTupleHashTable's entrysize (renamed to additionalsize) has been changed to only be about the additionally stored size. That size is only used for the initial sizing of the hash-table. Reviewed-By: Tomas Vondra Discussion: <20160727004333.r3e2k2y6fvk2ntup@alap3.anarazel.de> http://git.postgresql.org/pg/commitdiff/5dfc198146b49ce7ecc8a1fc9d5e171fb75f6ba5

Robert Haas pushed:

Tatsuo Ishii pushed:

Correctifs en attente

Michaël Paquier sent in a patch to use pg_ctl promote -w in TAP tests.

Michaël Paquier sent in a patch to fix some FSM corruption.

Pavel Stěhule sent in another revision of a patch to add \setfileref to psql.

Heikki Linnakangas sent in two more revisions of a patch to simplify the tape block format, and add pause/resume support for tapes.

Thomas Munro sent in another revision of a patch to enable DISTINCT with btree skip scan.

Mithun Cy sent in another revision of a patch to add some tests to cover hash_index.

Amit Langote sent in a patch to remove the int2vector operator hash infrastructure.

Haribabu Kommi sent in a patch to add a 64-bit (EUI-64) MAC address data type.

Dilip Kumar sent in a patch to enable scan key push down to heap.

Haribabu Kommi sent in a patch to add a pg_stat_sql system view.

Jim Nasby sent in a patch to add support for SRFs and returning composites from pl/tcl.

Robert Haas sent in another revision of a patch to bump up max-parallel-workers.

Amit Kapila sent in a patch to enables parallelism for btree scans.

Prabhat Kumar Sahu and Jeevan Chalke traded patches to do aggregate pushdown to a foreign server.

Aleksander Alekseev sent in a patch to fix some infelicities in pg_filedump.

Michaël Paquier sent in another revision of a patch to rename pg_xlog to pg_wal and two alternatives for renaming pg_clog: pg_transaction, and pg_xact.

Heikki Linnakangas sent in a patch to replace the polyphase merge algorithm with a simple balanced k-way merge.

Michaël Paquier and Heikki Linnakangas traded patches to add SCRAM auth.

Michaël Paquier sent in a patch to improve OOM handling in pg_locale.c.

Mithun Cy sent in another revision of a patch to implement failover on libpq connect level.

Etsuro Fujita sent in another revision of a patch to push down more full joins in postgres_fdw.

Peter Eisentraut sent in a patch to simplify internal archive version handling in pg_dump.

Etsuro Fujita sent in a patch to clarify the mention of self-joins in the DELETE documentation.

Pavel Stěhule sent in another revision of a patch to add xmltable().

Ashutosh Bapat sent in another revision of a patch to optimize partition-wise JOINs in tables with declarative partitions.

Michaël Paquier sent in a patch to improve the durability of pg_dump(all).

Stas Kelvich sent in a patch to fix some of the logging for 2PC.

Christoph Berg sent in a patch to choose a non-empty default log_line_prefix.

Peter Eisentraut sent in a patch to make getrusage() output a little more readable.

Heikki Linnakangas sent in another revision of a patch to support multi-dimensional arrays in PL/python and give a hint when [] is incorrectly used for a composite type in array.

Magnus Hagander sent in another revision of a patch to enable pg_basebackup to stream xlog to tar.

Jeff Janes sent in a patch to change the auth check in postgres_fdw to something a little stricter.

Jim Nasby sent in a patch to return a bitmask of NULL fields in a record.

par N Bougain le mardi 18 octobre 2016 à 20h51

lundi 10 octobre 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 9 octobre 2016

[ndt: Meetup à Nantes le 22 novembre : http://www.meetup.com/fr-FR/PostgreSQL-User-Group-Nantes/]

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en octobre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161009225109.GB15525@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Enforce a specific order for probing library loadability in pg_upgrade. pg_upgrade checks whether all the shared libraries used in the old cluster are also available in the new one by issuing LOAD for each library name. Previously, it cared not what order it did the LOADs in. Ideally it should not have to care, but currently the transform modules in contrib fail unless both the language and datatype modules they depend on are loaded first. A backend-side solution for that looks possible but probably not back-patchable, so as a stopgap measure, let's do the LOAD tests in order by library name length. That should fix the problem for reasonably-named transform modules, eg "hstore_plpython" will be loaded after both "hstore" and "plpython". (Yeah, it's a hack.) In a larger sense, having a predictable order of these probes is a good thing, since it will make upgrades predictably work or not work in the face of inter-library dependencies. Also, this patch replaces O(N^2) de-duplication logic with O(N log N) logic, which could matter in installations with very many databases. So I don't foresee reverting this even after we have a proper fix for the library-dependency problem. In passing, improve a couple of SQL queries used here. Per complaint from Andrew Dunstan that pg_upgrade'ing the transform contrib modules failed. Back-patch to 9.5 where transform modules were introduced. Discussion: <f7ac29f3-515c-2a44-21c5-ec925053265f@dunslane.net> http://git.postgresql.org/pg/commitdiff/83c2492002162bf79d2a0811bff5724e395909d7
  • Show a sensible value in pg_settings.unit for GUC_UNIT_XSEGS variables. Commit 88e982302 invented GUC_UNIT_XSEGS for min_wal_size and max_wal_size, but neglected to make it display sensibly in pg_settings.unit (by adding a case to the switch in GetConfigOptionByNum). Fix that, and adjust said switch to throw a run-time error the next time somebody forgets. In passing, avoid using a static buffer for the output string --- the rest of this function pstrdup's from a local buffer, and I see no very good reason why the units code should do it differently and less safely. Per report from Otar Shavadze. Back-patch to 9.5 where the new unit type was added. Report: <CAG-jOyA=iNFhN+yB4vfvqh688B7Tr5SArbYcFUAjZi=0Exp-Lg@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/6bc811c992a804bdb8d228ce0be9f0a8e7198df6
  • Convert contrib/hstore_plpython to not use direct linking to other modules. Previously, on most platforms, we allowed hstore_plpython's references to hstore and plpython to be unresolved symbols at link time, trusting the dynamic linker to resolve them when the module is loaded. This has a number of problems, the worst being that the dynamic linker does not know where the references come from and can do nothing but fail if those other modules haven't been loaded. We've more or less gotten away with that for the limited use-case of datatype transform modules, but even there, it requires some awkward hacks, most recently commit 83c249200. Instead, let's not treat these references as linker-resolvable at all, but use function pointers that are manually filled in by the module's _PG_init function. There are few enough contact points that this doesn't seem unmaintainable, at least for these use-cases. (Note that the same technique wouldn't work at all for decoupling from libpython itself, but fortunately that's just a standard shared library and can be linked to normally.) This is an initial patch that just converts hstore_plpython. If the buildfarm doesn't find any fatal problems, I'll work on the other transform modules soon. Tom Lane, per an idea of Andres Freund's. Discussion: <2652.1475512158@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/d51924be886c2a05e691fa05b16cb6b30ab8370f
  • Fix hstore_plpython for Python 3. In commit d51924be8, I overlooked the need to provide linkage for PLyUnicode_FromStringAndSize, because that's only used (and indeed only exists) in Python 3 builds. In light of the need to #if this item, rearrange the ordering of the code related to each function pointer, so as not to need more #if's than absolutely necessary. Per buildfarm. http://git.postgresql.org/pg/commitdiff/490ed1ebb9b26fb342a1e3240107092e9c5aec02
  • Improve (I hope) our autolocation of the Python shared library. Older versions of Python produce garbage (or at least useless) values of get_config_vars('LDLIBRARY'). Newer versions produce garbage (or at least useless) values of get_config_vars('SO'), which was defeating our configure logic that attempted to identify where the Python shlib really is. The net result, at least with a stock Python 3.5 installation on macOS, was that we were linking against a static library in the mistaken belief that it was a shared library. This managed to work, if you count statically absorbing libpython into plpython.so as working. But it no longer works as of commit d51924be8, because now we get separate static copies of libpython in plpython.so and hstore_plpython.so, and those can't interoperate on the same data. There are some other infelicities like assuming that nobody ever installs a private version of Python on a macOS machine. Hence, forget about looking in $python_configdir for the Python shlib; as far as I can tell no version of Python has ever put one there, and certainly no currently-supported version does. Also, rather than relying on get_config_vars('SO'), just try all the possibilities for shlib extensions. Also, rather than trusting Py_ENABLE_SHARED, believe we've found a shlib only if it has a recognized extension. Last, explicitly cope with the possibility that the shlib is really in /usr/lib and $python_libdir is a red herring --- this is the actual situation on older macOS, but we were only accidentally working with it. Discussion: <5300.1475592228@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/46ddbbb1177a7e6ce5a4fe0ef8fa8ac49f36d0bb
  • Avoid direct cross-module links in hstore_plperl and ltree_plpython, too. Just turning the crank on the project started in commit d51924be8. These cases turn out to be exact subsets of the boilerplate needed for hstore_plpython. Discussion: <2652.1475512158@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/eda04886c1e048d695728206504ab4198462168e
  • Huh, we do need to look in $python_configdir for the Python shlib. Debian does it that way, for no doubt what seems to them a good reason. Thanks to Aidan Van Dyk for confirmation. http://git.postgresql.org/pg/commitdiff/fc76259f5b8473dbd3d2009b0e4a267cf3a7e704
  • In python shlib probe, cater for OpenBSD, which omits the .so symlink. Most Unix-oid platforms provide a symlink "libfoo.so" -> "libfoo.so.n.n" to allow the linker to resolve a reference "-lfoo" to a version-numbered shared library. OpenBSD has apparently hacked ld(1) to do this internally, because there are no such symlinks to be found in their library directories. Tweak the new code in PGAC_CHECK_PYTHON_EMBED_SETUP to cope. Per buildfarm member curculio. http://git.postgresql.org/pg/commitdiff/ddd4f82cb6f65354776541dfac3bedf680e0e303
  • Try to fix python shlib probe for MinGW. Per discussion with Andrew Dunstan, it seems there are three peculiarities of the Python installation on MinGW (or at least, of the instance on buildfarm member frogmouth). First, the library name doesn't contain "2.7" but just "27". It looks like we can deal with that by consulting get_config_vars('VERSION') to see whether a dot should be used or not. Second, the library might be in c:/Windows/System32, analogously to it possibly being in /usr/lib on Unix-oid platforms. Third, it's apparently not standard to use the prefix "lib" on the file name. This patch will accept files with or without "lib", but the first part of that may well be dead code. http://git.postgresql.org/pg/commitdiff/11c0e743b67acc9a0406b6058ed24de5e5527cf0
  • Remove -Wl,-undefined,dynamic_lookup in macOS build. We don't need this anymore, and it prevents build-time error checking that's usually good to have, so remove it. Undoes one change of commit cac765820. Unfortunately, it's much harder to get a similar effect on other common platforms, because we don't want the linker to throw errors for symbols that will be resolved in the core backend. Only macOS and AIX expect the core backend executable to be available while linking loadable modules, so only these platforms can usefully throw errors for unresolved symbols at link time. Discussion: <2652.1475512158@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/bfe2e847811a4d0a46c8cdf16195c0e5929b6f83
  • Fix pg_dump to work against pre-9.0 servers again. getBlobs' queries for pre-9.0 servers were broken in two ways: the 7.x/8.x query uses DISTINCT so it can't have unspecified-type NULLs in the target list, and both that query and the 7.0 one failed to provide the correct output column labels, so that the subsequent code to extract data from the PGresult would fail. Back-patch to 9.6 where the breakage was introduced (by commit 23f34fa4b). Amit Langote and Tom Lane Discussion: <0a3e7a0e-37bd-8427-29bd-958135862f0a@lab.ntt.co.jp> http://git.postgresql.org/pg/commitdiff/4806f26f9e4273888cbf85f42b3bdc5e9d950ba6
  • Fix python shlib probe for Cygwin. On buildfarm member cockatiel, that library is in /usr/bin. (Possibly we should look at $PATH on this platform?) Per off-list report from Andrew Dunstan. http://git.postgresql.org/pg/commitdiff/17a3a1eb0efc5d84c81e46a26fe6bd21dbe90de9
  • libpqwalreceiver needs to link with libintl when using --enable-nls. The need for this was previously obscured even on picky platforms by the hack we used to support direct cross-module references in the transforms contrib modules. Now that that hack is gone, the undefined symbol is exposed, as reported by Robert Haas. Back-patch to 9.5 where we started to use -Wl,-undefined,dynamic_lookup. I'm a bit surprised that the older branches don't seem to contain any gettext references in this module, but since they don't fail at build time, they must not. (We might be able to get away with leaving this alone in 9.5/9.6, but I think it's cleaner if the reference gets resolved at link time.) Report: <CA+TgmoaHJKU5kcWZcYduATYVT7Mnx+8jUnycaYYL7OtCwCigug@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/8811f5d3a4a1ebf79ccb00da336d70041b003dd2
  • Fix two bugs in merging of inherited CHECK constraints. Historically, we've allowed users to add a CHECK constraint to a child table and then add an identical CHECK constraint to the parent. This results in "merging" the two constraints so that the pre-existing child constraint ends up with both conislocal = true and coninhcount > 0. However, if you tried to do it in the other order, you got a duplicate constraint error. This is problematic for pg_dump, which needs to issue separated ADD CONSTRAINT commands in some cases, but has no good way to ensure that the constraints will be added in the required order. And it's more than a bit arbitrary, too. The goal of complaining about duplicated ADD CONSTRAINT commands can be served if we reject the case of adding a constraint when the existing one already has conislocal = true; but if it has conislocal = false, let's just make the ADD CONSTRAINT set conislocal = true. In this way, either order of adding the constraints has the same end result. Another problem was that the code allowed creation of a parent constraint marked convalidated that is merged with a child constraint that is !convalidated. In this case, an inheritance scan of the parent table could emit some rows violating the constraint condition, which would be an unexpected result given the marking of the parent constraint as validated. Hence, forbid merging of constraints in this case. (Note: valid child and not-valid parent seems fine, so continue to allow that.) Per report from Benedikt Grundmann. Back-patch to 9.2 where we introduced possibly-not-valid check constraints. The second bug obviously doesn't apply before that, and I think the first doesn't either, because pg_dump only gets into this situation when dealing with not-valid constraints. Report: <CADbMkNPT-Jz5PRSQ4RbUASYAjocV_KHUWapR%2Bg8fNvhUAyRpxA%40mail.gmail.com> Discussion: <22108.1475874586@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/e55a946a81c6648c5ff3a0ccdda242f639e33c6f
  • Fix incorrect handling of polymorphic aggregates used as window functions. The transfunction was told that its first argument and result were of the window function output type, not the aggregate state type. This'd only matter if the transfunction consults get_fn_expr_argtype, which typically only polymorphic functions would do. Although we have several regression tests around polymorphic aggs, none of them detected this mistake --- in fact, they still didn't fail when I injected the same mistake into nodeAgg.c. So add some more tests covering both plain agg and window-function-agg cases. Per report from Sebastian Luque. Back-patch to 9.6 where the error was introduced (by sloppy refactoring in commit 804163bc2, looks like). Report: <87int2qkat.fsf@gmail.com> http://git.postgresql.org/pg/commitdiff/ac4a9d92fcb6869e757cc729dca2ca5ccf94b185
  • Use unnamed POSIX semaphores, if available, on Linux and FreeBSD. We've had support for using unnamed POSIX semaphores instead of System V semaphores for quite some time, but it was not used by default on any platform. Since many systems have rather small limits on the number of SysV semaphores allowed, it seems desirable to switch to POSIX semaphores where they're available and don't create performance or kernel resource problems. Experimentation by me shows that unnamed POSIX semaphores are at least as good as SysV semaphores on Linux, and we previously had a report from Maksym Sobolyev that FreeBSD is significantly worse with SysV semaphores than POSIX ones. So adjust those two platforms to use unnamed POSIX semaphores, if configure can find the necessary library functions. If this goes well, we may switch other platforms as well, but it would be advisable to test them individually first. It's not currently contemplated that we'd encourage users to select a semaphore API for themselves, but anyone who wants to experiment can add PREFERRED_SEMAPHORES=UNNAMED_POSIX (or NAMED_POSIX, or SYSV) to their configure command line to do so. I also tweaked configure to report which API it's selected, mainly so that we can tell that from buildfarm reports. I did not touch the user documentation's discussion about semaphores; that will need some adjustment once the dust settles. Discussion: <8536.1475704230@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/ecb0d20a9d2e09b7112d3b192047f711f9ff7e59

Stephen Frost pushed:

  • Fix RLS with COPY (col1, col2) FROM tab Attempting to COPY a subset of columns from a table with RLS enabled would fail due to an invalid query being constructed (using a single ColumnRef with the list of fields to exact in 'fields', but that's for the different levels of an indirection for a single column, not for specifying multiple columns). Correct by building a ColumnRef and then RestTarget for each column being requested and then adding those to the targetList for the select query. Include regression tests to hopefully catch if this is broken again in the future. Patch-By: Adam Brightwell Reviewed-By: Michael Paquier http://git.postgresql.org/pg/commitdiff/814b9e9b8edf36cac65e0d8fcef17e50a04b1617

Andres Freund pushed:

  • Correct logical decoding restore behaviour for subtransactions. Before initializing iteration over a subtransaction's changes, the last few changes were not spilled to disk. That's correct if the transaction didn't spill to disk, but otherwise... This bug can lead to missed or misorderd subtransaction contents when they were spilled to disk. Move spilling of the remaining in-memory changes to ReorderBufferIterTXNInit(), where it can easily be applied to the top transaction and, if present, subtransactions. Since this code had too many bugs already, noticeably increase test coverage. Fixes: #14319 Reported-By: Huan Ruan Discussion: <20160909012610.20024.58169@wrigleys.postgresql.org> Backport: 9,4-, where logical decoding was added http://git.postgresql.org/pg/commitdiff/61633f79048d040aefeed68dcac0e0b996a06189
  • Fix fallback implementation of pg_atomic_write_u32(). I somehow had assumed that in the spinlock (in turn possibly using semaphores) based fallback atomics implementation 32 bit writes could be done without a lock. As far as the write goes that's correct, since postgres supports only platforms with single-copy atomicity for aligned 32bit writes. But writing without holding the spinlock breaks read-modify-write operations like pg_atomic_compare_exchange_u32(), since they'll potentially "miss" a concurrent write, which can't happen in actual hardware implementations. In 9.6+ when using the fallback atomics implementation this could lead to buffer header locks not being properly marked as released, and potentially some related state corruption. I don't see a related danger in 9.5 (earliest release with the API), because pg_atomic_write_u32() wasn't used in a concurrent manner there. The state variable of local buffers, before this change, were manipulated using pg_atomic_write_u32(), to avoid unnecessary synchronization overhead. As that'd not be the case anymore, introduce and use pg_atomic_unlocked_write_u32(), which does not correctly interact with RMW operations. This bug only caused issues when postgres is compiled on platforms without atomics support (i.e. no common new platform), or when compiled with --disable-atomics, which explains why this wasn't noticed in testing. Reported-By: Tom Lane Discussion: <14947.1475690465@sss.pgh.pa.us> Backpatch: 9.5-, where the atomic operations API was introduced. http://git.postgresql.org/pg/commitdiff/b0779abb3add11d4dd745779dd81ea8ecdd00a1d

Heikki Linnakangas pushed:

  • Update comment. mergepreread()/mergeprereadone() don't exist anymore, the function that does roughly the same is now called mergereadnext(). http://git.postgresql.org/pg/commitdiff/c86c2d9d57c1e6c5fec860873734cccaf31df5b0
  • Fix another outdated comment. Preloading is done by logtape.c now. http://git.postgresql.org/pg/commitdiff/d4fca5e6c7363ba6ee4de7b8d72d68064fa864ca
  • Change the way pre-reading in external sort's merge phase works. Don't pre-read tuples into SortTuple slots during merge. Instead, use the memory for larger read buffers in logtape.c. We're doing the same number of READTUP() calls either way, but managing the pre-read SortTuple slots is much more complicated. Also, the on-tape representation is more compact than SortTuples, so we can fit more pre-read tuples into the same amount of memory this way. And we have better cache-locality, when we use just a small number of SortTuple slots. Now that we only hold one tuple from each tape in the SortTuple slots, we can greatly simplify the "batch memory" management. We now maintain a small set of fixed-sized slots, to hold the tuples, and fall back to palloc() for larger tuples. We use this method during all merge phases, not just the final merge, and also when randomAccess is requested, and also in the TSS_SORTEDONTAPE case. In other words, it's used whenever we do an external sort. Reviewed by Peter Geoghegan and Claudio Freire. Discussion: <CAM3SWZTpaORV=yQGVCG8Q4axcZ3MvF-05xe39ZvORdU9JcD6hQ@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/e94568ecc10f2638e542ae34f2990b821bbf90ac
  • Fix excessive memory consumption in the new sort pre-reading code. LogicalTapeRewind() should not allocate large read buffer, if the tape is completely empty. The calling code relies on that, for its calculation of how much memory to allocate for the read buffers. That lead to massive overallocation of memory, if maxTapes was high, but only a few tapes were actually used. Reported by Tomas Vondra Discussion: <7303da46-daf7-9c68-3cc1-9f83235cf37e@2ndquadrant.com> http://git.postgresql.org/pg/commitdiff/b56fb691b0033f9b86e0552bd5adfd485f05eef6
  • Disable synchronous commits in pg_rewind. If you point pg_rewind to a server that is using synchronous replication, with "pg_rewind --source-server=...", and the replication is not working for some reason, pg_rewind will get stuck because it creates a temporary table, which needs to be replicated. You could call broken replication a pilot error, but pg_rewind is often used in special circumstances, when there are changes to the replication setup. We don't do any "real" updates, and we don't care about fsyncing or replicating the operations on the temporary tables, so fix that by setting synchronous_commit off. Michael Banck, Michael Paquier. Backpatch to 9.5, where pg_rewind was introduced. Discussion: <20161005143938.GA12247@nighthawk.caipicrew.dd-dns.de> http://git.postgresql.org/pg/commitdiff/d7eb76b908a3aac68c23c7d3553c65c80e92b823
  • Don't share SSL_CTX between libpq connections. There were several issues with the old coding: 1. There was a race condition, if two threads opened a connection at the same time. We used a mutex around SSL_CTX_* calls, but that was not enough, e.g. if one thread SSL_CTX_load_verify_locations() with one path, and another thread set it with a different path, before the first thread got to establish the connection. 2. Opening two different connections, with different sslrootcert settings, seemed to fail outright with "SSL error: block type is not 01". Not sure why. 3. We created the SSL object, before calling SSL_CTX_load_verify_locations and SSL_CTX_use_certificate_chain_file on the SSL context. That was wrong, because the options set on the SSL context are propagated to the SSL object, when the SSL object is created. If they are set after the SSL object has already been created, they won't take effect until the next connection. (This is bug #14329) At least some of these could've been fixed while still using a shared context, but it would've been more complicated and error-prone. To keep things simple, let's just use a separate SSL context for each connection, and accept the overhead. Backpatch to all supported versions. Report, analysis and test case by Kacper Zuk. Discussion: <20160920101051.1355.79453@wrigleys.postgresql.org> http://git.postgresql.org/pg/commitdiff/8bb14cdd33deecc6977cf5638f73e80f76ab658b
  • Clear OpenSSL error queue after failed X509_STORE_load_locations() call. Leaving the error in the error queue used to be harmless, because the X509_STORE_load_locations() call used to be the last step in initialize_SSL(), and we would clear the queue before the next SSL_connect() call. But previous commit moved things around. The symptom was that if a CRL file was not found, and one of the subsequent initialization steps, like loading the client certificate or private key, failed, we would incorrectly print the "no such file" error message from the earlier X509_STORE_load_locations() call as the reason. Backpatch to all supported versions, like the previous patch. http://git.postgresql.org/pg/commitdiff/275bf98601b230e530003ef20193d095b9309c24
  • Don't allow both --source-server and --source-target args to pg_rewind. They are supposed to be mutually exclusive, but there was no check for that. Michael Banck Discussion: <20161007103414.GD12247@nighthawk.caipicrew.dd-dns.de> http://git.postgresql.org/pg/commitdiff/0d4d7d61850f4f4bc5f6fd0b7a9adb70232aed61
  • Make TAP test suites to work, when @INC does not contain current dir. Recent Perl and/or new Linux distributions are starting to remove "." from the @INC list by default. That breaks pg_rewind and ssl test suites, which use helper perl modules that reside in the same directory. To fix, add the current source directory explicitly to prove's include dir. The vcregress.pl script probably also needs something like this, but I wasn't able to remove '.' from @INC on Windows to test this, and don't want to try doing that blindly. Discussion: <20160908204529.flg6nivjuwp5vaoy@alap3.anarazel.de> http://git.postgresql.org/pg/commitdiff/d668b03378c28b927d0ba458681ca1b4c1e18e53
  • Remove bogus mapping from UTF-8 to SJIS conversion table. 0xc19c is not a valid UTF-8 byte sequence. It doesn't do any harm, AFAICS, but it's surely not intentional. No backpatching though, just to be sure. In the passing, also add a file header comment to the file, like the UCS_to_SJIS.pl script would produce. (The file was originally created with UCS_to_SJIS.pl, but has been modified by hand since then. That's questionable, but I'll leave fixing that for later.) Kyotaro Horiguchi Discussion: <20160907.155050.233844095.horiguchi.kyotaro@lab.ntt.co.jp> http://git.postgresql.org/pg/commitdiff/0aec7f9aec8b828e074b8f2f3cbea2ec3e5c0209

Robert Haas pushed:

Correctifs en attente

Fabien COELHO sent in a patch to allow backslash continuations in \set expressions in pgbench.

Amit Langote sent in a patch to un-include access/heapam.h.

Michaël Paquier sent in another revision of a patch to rename pg_xlog to pg_wal and rename pg_clog to pg_transaction.

Anastasia Lubennikova sent in two more revisions of a patch to add covering + unique indexes.

Julien Rouhaud sent in another revision of a patch to add the parallel worker class and change max_worker_processes default to 16 and max_parallel_workers to 8.

Thomas Munro sent in another revision of a patch to add condition variables.

Thomas Munro sent in a patch to create a DSA area to provide work space for parallel execution.

Rushabh Lathia sent in a patch to implement Gather Merge.

Michaël Paquier sent in another revision of a patch to fix hot standby checkpoints.

Thomas Munro sent in two more revisions of a patch to add dynamic shared memory areas.

Tomas Vondra sent in another revision of a patch to add two slab-like memory allocators.

Takayuki Tsunakawa sent in another revision of a patch to close listen ports earlier in the shutdown process.

Thomas Munro sent in two revisions of a patch to use DSA memory so that hash tables can be shared by multiple backends and yet grow dynamically. Development name: "DHT".

Etsuro Fujita sent in three more revisions of a patch to fix invalidation for prepared statements with FDWs.

Amit Langote sent in three more revisions of a patch to implement declarative partitioning.

Fabien COELHO sent in another revision of a patch to add more operators and functions to pgbench.

Robert Haas sent in a patch to improve on pgrminclude / pgcompinclude.

Daniel Gustafsson sent in a WIP patch to add Secure Transport support as OpenSSL alternative on macOS.

Vitaly Burovoy sent in a patch to fix wording in the docs on parallel queries.

Franck Verrot sent in another revision of a patch to mntion column name in error messages.

Ashutosh Bapat sent in a patch to pass input_rel->relids to fetch_upper_rel() in create_grouping_path().

Heikki Linnakangas sent in a patch to always use 2048 bit DH parameters for OpenSSL ephemeral DH ciphers.

Dilip Kumar sent in a patch to implement parallel bitmap heap scan.

Pavan Deolasee sent in a patch to prevent certain types of FSM corruption.

Gilles Darold sent in another revision of a patch to implement a pg_current_logfile() function.

Peter Geoghegan sent in another revision of a patch to add parallel tuplesort for parallel B-Tree index creation.

Jeff Janes sent in a patch to add a pgbench transaction function.

Pavel Stěhule sent in a patch to add a parallel status to psql's \df+.

Pavel Stěhule sent in another revision of a patch to add \setfileref to psql.

par N Bougain le lundi 10 octobre 2016 à 22h00

Nouvelles hebdomadaires de PostgreSQL - 2 octobre 2016

PostgreSQL 9.6 est disponible ! https://www.postgresql.org/docs/current/static/release-9-6.html

Les nouveautés des produits dérivés

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20161002231634.GB1769@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Tom Lane pushed:

  • Refer to OS X as "macOS", except for the port name which is still "darwin". We weren't terribly consistent about whether to call Apple's OS "OS X" or "Mac OS X", and the former is probably confusing to people who aren't Apple users. Now that Apple has rebranded it "macOS", follow their lead to establish a consistent naming pattern. Also, avoid the use of the ancient project name "Darwin", except as the port code name which does not seem desirable to change. (In short, this patch touches documentation and comments, but no actual code.) I didn't touch contrib/start-scripts/osx/, either. I suspect those are obsolete and due for a rewrite, anyway. I dithered about whether to apply this edit to old release notes, but those were responsible for quite a lot of the inconsistencies, so I ended up changing them too. Anyway, Apple's being ahistorical about this, so why shouldn't we be? http://git.postgresql.org/pg/commitdiff/da6c4f6ca88df346573bdada2aa2544510bf167e
  • Document has_type_privilege(). Evidently an oversight in commit 729205571. Back-patch to 9.2 where privileges for types were introduced. Report: <20160922173517.8214.88959@wrigleys.postgresql.org> http://git.postgresql.org/pg/commitdiff/a4afb2b5c0b409bb175c20104b2ae9d47cf71be6
  • Replace the built-in GIN array opclasses with a single polymorphic opclass. We had thirty different GIN array opclasses sharing the same operators and support functions. That still didn't cover all the built-in types, nor did it cover arrays of extension-added types. What we want is a single polymorphic opclass for "anyarray". There were two missing features needed to make this possible: 1. We have to be able to declare the index storage type as ANYELEMENT when the opclass is declared to index ANYARRAY. This just takes a few more lines in index_create(). Although this currently seems of use only for GIN, there's no reason to make index_create() restrict it to that. 2. We have to be able to identify the proper GIN compare function for the index storage type. This patch proceeds by making the compare function optional in GIN opclass definitions, and specifying that the default btree comparison function for the index storage type will be looked up when the opclass omits it. Again, that seems pretty generically useful. Since the comparison function lookup is done in initGinState(), making use of the second feature adds an additional cache lookup to GIN index access setup. It seems unlikely that that would be very noticeable given the other costs involved, but maybe at some point we should consider making GinState data persist longer than it now does --- we could keep it in the index relcache entry, perhaps. Rather fortuitously, we don't seem to need to do anything to get this change to play nice with dump/reload or pg_upgrade scenarios: the new opclass definition is automatically selected to replace existing index definitions, and the on-disk data remains compatible. Also, if a user has created a custom opclass definition for a non-builtin type, this doesn't break that, since CREATE INDEX will prefer an exact match to opcintype over a match to ANYARRAY. However, if there's anyone out there with handwritten DDL that explicitly specifies _bool_ops or one of the other replaced opclass names, they'll need to adjust that. Tom Lane, reviewed by Enrique Meneses Discussion: <14436.1470940379@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/fdc9186f7ed1ead827509584f3b763f8dc332c43
  • Fix newly-introduced issues in pgbench. The result of FD_ISSET() doesn't necessarily fit in a bool, though assigning it to one might accidentally work depending on platform and which socket FD number is being inquired of. Rewrite to test it with if(), rather than making any specific assumption about the result width, to match the way every other such call in PG is written. Don't break out of the input_mask-filling loop after finding the first client that we're waiting for results from. That mostly breaks parallel query management. Also, if we choose not to call select(), be sure to clear out any bits the mask-filling loop might have set, so that we don't accidentally call doCustom for clients we don't know have input. Doing so would likely be harmless, but it's a waste of cycles and doesn't seem to be intended. Make this_usec wide enough. (Yeah, the value would usually fit in an int, but then why are we using int64 everywhere else?) Minor cosmetic adjustments, mostly comment improvements. Problems introduced by commit 12788ae49. The first issue was discovered by buildfarm testing, the others by code review. http://git.postgresql.org/pg/commitdiff/9779bda86c026e645773a3308f9169c7c0791f7c
  • Improve contrib/cube's handling of zero-D cubes, infinities, and NaNs. It's always been possible to create a zero-dimensional cube by converting from a zero-length float8 array, but cube_in failed to accept the '()' representation that cube_out produced for that case, resulting in a dump/reload hazard. Make it accept the case. Also fix a couple of other places that didn't behave sanely for zero-dimensional cubes: cube_size would produce 1.0 when surely the answer should be 0.0, and g_cube_distance risked a divide-by-zero failure. Likewise, it's always been possible to create cubes containing float8 infinity or NaN coordinate values, but cube_in couldn't parse such input, and cube_out produced platform-dependent spellings of the values. Convert them to use float8in_internal and float8out_internal so that the behavior will be the same as for float8, as we recently did for the core geometric types (cf commit 50861cd68). As in that commit, I don't pretend that this patch fixes all insane corner-case behaviors that may exist for NaNs, but it's a step forward. (This change allows removal of the separate cube_1.out and cube_3.out expected-files, as the platform dependency that previously required them is now gone: an underflowing coordinate value will now produce an error not plus or minus zero.) Make errors from cube_in follow project conventions as to spelling ("invalid input syntax for cube" not "bad cube representation") and errcode (INVALID_TEXT_REPRESENTATION not SYNTAX_ERROR). Also a few marginal code cleanups and comment improvements. Tom Lane, reviewed by Amul Sul Discussion: <15085.1472494782@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/f31a931fade868d788ef4480c59753a2d5059246
  • Redesign parallel dump/restore's wait-for-workers logic. The ListenToWorkers/ReapWorkerStatus APIs were messy and hard to use. Instead, make DispatchJobForTocEntry register a callback function that will take care of state cleanup, doing whatever had been done by the caller of ReapWorkerStatus in the old design. (This callback is essentially just the old mark_work_done function in the restore case, and a trivial test for worker failure in the dump case.) Then we can have ListenToWorkers call the callback immediately on receipt of a status message, and return the worker to WRKR_IDLE state; so the WRKR_FINISHED state goes away. This allows us to design a unified wait-for-worker-messages loop: WaitForWorkers replaces EnsureIdleWorker and EnsureWorkersFinished as well as the mess in restore_toc_entries_parallel. Also, we no longer need the fragile API spec that the caller of DispatchJobForTocEntry is responsible for ensuring there's an idle worker, since DispatchJobForTocEntry can just wait until there is one. In passing, I got rid of the ParallelArgs struct, which was a net negative in terms of notational verboseness, and didn't seem to be providing any noticeable amount of abstraction either. Tom Lane, reviewed by Kevin Grittner Discussion: <1188.1464544443@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/b7b8cc0cfcf1c956b752f3e25894f9ad607583b7
  • Rationalize parallel dump/restore's handling of worker cmd/status messages. The existing APIs for creating and parsing command and status messages are rather messy; for example, archive-format modules have to provide code for constructing command messages, which is entirely pointless since the code to read them is hard-wired in WaitForCommands() and hence no format-specific variation is actually possible. But there's little foreseeable reason to need format-specific variation anyway. The situation for status messages is no better; at least those are both constructed and parsed by format-specific code, but said code is quite redundant since there's no actual need for format-specific variation. To add insult to injury, the first API involves returning pointers to static buffers, which is bad, while the second involves returning pointers to malloc'd strings, which is safer but randomly inconsistent. Hence, get rid of the MasterStartParallelItem and MasterEndParallelItem APIs, and instead write centralized functions that construct and parse command and status messages. If we ever do need more flexibility, these functions can be the standard implementations of format-specific callback methods, but that's a long way off if it ever happens. Tom Lane, reviewed by Kevin Grittner Discussion: <17340.1464465717@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/fb03d08a89e81a68585f17fd8e7f21c618f4e851
  • Make struct ParallelSlot private within pg_dump/parallel.c. The only field of this struct that other files have any need to touch is the pointer to the TocEntry a worker is working on. (Well, pg_backup_archiver.c is actually looking at workerStatus too, but that can be finessed by specifying that the TocEntry pointer is NULL for a non-busy worker.) Hence, move out the TocEntry pointers to a separate array within struct ParallelState, and then we can make struct ParallelSlot private. I noted the possibility of this previously, but hadn't got round to actually doing it. Discussion: <1188.1464544443@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/0109ab27609c0d58c1eddc6b799077d0968083de
  • Disallow pushing volatile quals past set-returning functions. Pushing an upper-level restriction clause into an unflattened subquery-in-FROM is okay when the subquery contains no SRFs in its targetlist, or when it does but the SRFs are unreferenced by the clause *and the clause is not volatile*. Otherwise, we're changing the number of times the clause is evaluated, which is bad for volatile quals, and possibly changing the result, since a volatile qual might succeed for some SRF output rows and not others despite not referencing any of the changing columns. (Indeed, if the clause is something like "random() > 0.5", the user is probably expecting exactly that behavior.) We had most of these restrictions down, but not the one about the upper clause not being volatile. Fix that, and add a regression test to illustrate the expected behavior. Although this is definitely a bug, it doesn't seem like back-patch material, since possibly some users don't realize that the broken behavior is broken and are relying on what happens now. Also, while the added test is quite cheap in the wake of commit a4c35ea1c, it would be much more expensive (or else messier) in older branches. Per report from Tom van Tilburg. Discussion: <CAP3PPDiucxYCNev52=YPVkrQAPVF1C5PFWnrQPT7iMzO1fiKFQ@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/72daabc7a3e75788df862104b8f723513c2471ae
  • Make to_timestamp() and to_date() range-check fields of their input. Historically, something like to_date('2009-06-40','YYYY-MM-DD') would return '2009-07-10' because there was no prohibition on out-of-range month or day numbers. This has been widely panned, and it also turns out that Oracle throws an error in such cases. Since these functions are nominally Oracle-compatibility features, let's change that. There's no particular restriction on year (modulo the fact that the scanner may not believe that more than 4 digits are year digits, a matter to be addressed separately if at all). But we now check month, day, hour, minute, second, and fractional-second fields, as well as day-of-year and second-of-day fields if those are used. Currently, no checks are made on ISO-8601-style week numbers or day numbers; it's not very clear what the appropriate rules would be there, and they're probably so little used that it's not worth sweating over. Artur Zakirov, reviewed by Amul Sul, further adjustments by me Discussion: <1873520224.1784572.1465833145330.JavaMail.yahoo@mail.yahoo.com> See-Also: <57786490.9010201@wars-nicht.de> http://git.postgresql.org/pg/commitdiff/d3cd36a133d96ad5578b6c10279b55fd5b538093
  • Rationalize format-picture caching logic in formatting.c. Add a validity flag to DCHCacheEntry and NUMCacheEntry entries, and do not set it true until after we've parsed the supplied format string. This allows dealing with possible errors while parsing the format without the baroque hack that was there before (which only covered errors within NUMDesc_prepare, anyway). We can get rid of the PG_TRY in NUMDesc_prepare, as well as last_NUMCacheEntry and NUM_cache_remove. (Essentially, this reverts commit ff783fbae in favor of a less fragile solution; the problems with that approach are well illustrated by later hacking such as 55f927a46.) In passing, define the size of these caches as DCH_CACHE_ENTRIES not DCH_CACHE_FIELDS + 1 (whoever thought that was a good definition?) and likewise for the NUM cache. Also const-ify format string parameters where convenient, and merge duplicated cache lookup logic. This is primarily driven by a proposed patch from Artur Zakirov, which introduced some ereport's into format string parsing for the datetime case. He proposed preventing the creation of invalid cache entries by parsing the format string first into a local-variable array, and then copying that to a cache entry. That seemed a bit ugly to me, and anyway randomly different from the way the identical problem had been solved for the numeric case. Let's make the two sets of code more similar not less so. I'm not sure whether we'll adopt the new error conditions Artur proposes, but this patch seems like good code cleanup and future-proofing in any case. The existing code is critically (and undocumented-ly) dependent on no elog being thrown out of several nontrivial functions, which is trouble waiting to happen, though it doesn't seem to be actively broken today. Discussion: <b2a39359-3282-b402-f4a3-057aae500ee7@postgrespro.ru> http://git.postgresql.org/pg/commitdiff/83bed06be4e808f3da30f99b6c91e9efda3961ad
  • Allow contrib/file_fdw to read from a program, like COPY FROM PROGRAM. This patch just exposes COPY's FROM PROGRAM option in contrib/file_fdw. There don't seem to be any security issues with that that are any worse than what already exist with file_fdw and COPY; as in the existing cases, only superusers are allowed to control what gets executed. A regression test case might be nice here, but choosing a 100% portable command to run is hard. (We haven't got a test for COPY FROM PROGRAM itself, either.) Corey Huinker and Adam Gomaa, reviewed by Amit Langote Discussion: <CADkLM=dGDGmaEiZ=UDepzumWg-CVn7r8MHPjr2NArj8S3TsROQ@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/8e91e12bc3af85ba2287866669268f6825d2cc03
  • Fix multiple portability issues in pg_upgrade's rewriteVisibilityMap(). This is new code in 9.6, and evidently we missed out testing it as thoroughly as it should have been. Bugs fixed here: 1. Use binary not text mode to open the files on Windows. Before, if the visibility map chanced to contain two bytes that looked like \r\n, Windows' read() would convert that to \n, which both corrupts the map data and causes the file to look shorter than it should. Unless you were *very* unlucky and had an exact multiple of 8K such occurrences in each VM file, this would cause pg_upgrade to report a failure, though with a rather obscure error message. 2. The code for copying rebuilt bytes into the output was simply wrong. It chanced to work okay on little-endian machines but would emit the bytes in the wrong order on big-endian, leading to silent corruption of the visibility map data. 3. The code was careless about alignment of the working buffers. Given all three of an alignment-picky architecture, a compiler that chooses to put the new_vmbuf[] local variable at an odd starting address, and a checksum-enabled database, pg_upgrade would dump core. Point one was reported by Thomas Kellerer, the other two detected by code-reading. Point two is much the nastiest of these issues from an impact standpoint, though fortunately it affects only a minority of users. The Windows issue will definitely bite people, but it seems quite unlikely that there would be undetected corruption from that. In addition, I failed to resist the temptation to do some minor cosmetic adjustments, mostly improving the comments. It would be a good idea to try to improve the error reporting here, but that seems like material for a separate patch. ------- http://git.postgresql.org/pg/commitdiff/5afcd2aa740f87fa6cec7c6693a4891b9df2b67d
  • Improve error reporting in pg_upgrade's file copying/linking/rewriting. The previous design for this had copyFile(), linkFile(), and rewriteVisibilityMap() returning strerror strings, with the caller producing one-size-fits-all error messages based on that. This made it impossible to produce messages that described the failures with any degree of precision, especially not short-read problems since those don't set errno at all. Since pg_upgrade has no intention of continuing after any error in this area, let's fix this by just letting these functions call pg_fatal() for themselves, making it easy for each point of failure to have a suitable error message. Taking this approach also allows dropping cleanup code that was unnecessary and was often rather sloppy about preserving errno. To not lose relevant info that was reported before, pass in the schema name and table name of the current table so that they can be included in the error reports. An additional problem was the use of getErrorText(), which was flat out wrong for all but a couple of call sites, because it unconditionally did "_dosmaperr(GetLastError())" on Windows. That's only appropriate when reporting an error from a Windows-native API, which only a couple of the callers were actually doing. Thus, even the reported strerror string would be unrelated to the actual failure in many cases on Windows. To fix, get rid of getErrorText() altogether, and just have call sites do strerror(errno) instead, since that's the way all the rest of our frontend programs do it. Add back the _dosmaperr() calls in the two places where that's actually appropriate. In passing, make assorted messages hew more closely to project style guidelines, notably by removing initial capitals in not-complete-sentence primary error messages. (I didn't make any effort to clean up places I didn't have another reason to touch, though.) Per discussion of a report from Thomas Kellerer. Back-patch to 9.6, but no further; given the relative infrequency of reports of problems here, it's not clear it's worth adapting the patch to older branches. Patch by me, but with credit to Alvaro Herrera for spotting the issue with getErrorText's misuse of _dosmaperr(). Discussion: <nsjrbh$8li$1@blaine.gmane.org> http://git.postgresql.org/pg/commitdiff/f002ed2b8e45286fe73a36e119cba2016ea0de19
  • Fix misplacement of submake-generated-headers prerequisites. The sequence "configure; cd src/pl/plpython; make -j" failed due to trying to compile plpython's .o files before the generated headers finished building. (This is an important real-world case, since it's the typical second step when building both plpython2 and plpython3.) This happens because the submake-generated-headers target is not placed in a way to make it a prerequisite to compiling the .o files. Fix that. Checking other uses of submake-generated-headers, I noted that the one attached to pg_regress was similarly misplaced; but it's actually not needed at all for pg_regress.o, rather regress.o, so move it to be a prerequisite of that. Back-patch to 9.6 where submake-generated-headers was introduced (by commit 548af97fc). It's not immediately clear to me why the previous coding didn't have the same issue; but since we've not had field reports of plpython make failing, leave it alone in the older branches. Pavel Raiskup and Tom Lane Discussion: <1925924.izSMJEZO3x@unused-4-107.brq.redhat.com> http://git.postgresql.org/pg/commitdiff/7107d58ec5a3c45967e77525809612a5f89b97f3
  • Fix misstatement in comment in Makefile.shlib. There is no need for "all: all-lib" to be placed before inclusion of Makefile.shlib. Makefile.global is what ensures that "all" is the default target, and we already document that that has to be included first. Per comment from Pavel Raiskup. Discussion: <1925924.izSMJEZO3x@unused-4-107.brq.redhat.com> http://git.postgresql.org/pg/commitdiff/ea046f08d1bcee56c3bedfa16a05c38a0515b41d
  • Copy-editing for contrib/pg_visibility documentation. Add omitted names for some function parameters. Fix some minor grammatical issues. http://git.postgresql.org/pg/commitdiff/33596edf09516a7cab65914e16cfd6adf9fc55d1
  • Fix bugs in contrib/pg_visibility. collect_corrupt_items() failed to initialize tuple.t_self. While HeapTupleSatisfiesVacuum() doesn't actually use that value, it does Assert that it's valid, so that the code would dump core if ip_posid chanced to be zero. (That's somewhat unlikely, which probably explains how this got missed. In any case it wouldn't matter for field use.) Also, collect_corrupt_items was returning the wrong TIDs, that is the contents of t_ctid rather than the tuple's own location. This would be the same thing in simple cases, but it could be wrong if, for example, a past update attempt had been rolled back, leaving a live tuple whose t_ctid doesn't point at itself. Also, in pg_visibility(), guard against trying to read a page past the end of the rel. The VM code handles inquiries beyond the end of the map by silently returning zeroes, and it seems like we should do the same thing here. I ran into the assertion failure while using pg_visibility to check pg_upgrade's behavior, and then noted the other problems while reading the code. Report: <29043.1475288648@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/9a109452da1b923e183f20fcf5516984d448ece9
  • Do ClosePostmasterPorts() earlier in SubPostmasterMain(). In standard Unix builds, postmaster child processes do ClosePostmasterPorts immediately after InitPostmasterChild, that is almost immediately after being spawned. This is important because we don't want children holding open the postmaster's end of the postmaster death watch pipe. However, in EXEC_BACKEND builds, SubPostmasterMain was postponing this responsibility significantly, in order to make it slightly more convenient to pass the right flag value to ClosePostmasterPorts. This is bad, particularly seeing that process_shared_preload_libraries() might invoke nearly-arbitrary code. Rearrange so that we do it as soon as we've fetched the socket FDs via read_backend_variables(). Also move the comment explaining about randomize_va_space to before the call of PGSharedMemoryReAttach, which is where it's relevant. The old placement was appropriate when the reattach happened inside CreateSharedMemoryAndSemaphores, but that was a long time ago. Back-patch to 9.3; the patch doesn't apply cleanly before that, and it doesn't seem worth a lot of effort given that we've had no actual field complaints traceable to this. Discussion: <4157.1475178360@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/3b90e38c5d592ea8ec8236287dd5c749fc041728
  • Avoid leaking FDs after an fsync failure. Fixes errors introduced in commit bc34223bc, as detected by Coverity. In passing, report ENOSPC for a short write while padding a new wal file in open_walfile, make certain that close_walfile closes walfile in all cases, and improve a couple of comments. Michael Paquier and Tom Lane http://git.postgresql.org/pg/commitdiff/728ceba938dfadb204a4854bb76ae3b11b635401
  • Add ALTER EXTENSION ADD/DROP ACCESS METHOD, and use it in pg_upgrade. Without this, an extension containing an access method is not properly dumped/restored during pg_upgrade --- the AM ends up not being a member of the extension after upgrading. Another oversight in commit 473b93287, reported by Andrew Dunstan. Report: <f7ac29f3-515c-2a44-21c5-ec925053265f@dunslane.net> http://git.postgresql.org/pg/commitdiff/e8bdee2770ff52aab208bc6f8965a4a01979d0aa

Peter Eisentraut pushed:

Ãlvaro Herrera pushed:

Heikki Linnakangas pushed:

  • Refactor script execution state machine in pgbench. The doCustom() function had grown into quite a mess. Rewrite it, in a more explicit state machine style, for readability. This also fixes one minor bug: if a script consisted entirely of meta commands, doCustom() never returned to the caller, so progress reports with the -P option were not printed. I don't want to backpatch this refactoring, and the bug is quite insignificant, so only commit this to master, and leave the bug unfixed in back-branches. Review and original bug report by Fabien Coelho. Discussion: <alpine.DEB.2.20.1607090850120.3412@sto> http://git.postgresql.org/pg/commitdiff/12788ae49e1933f463bc59a6efe46c4a01701b76
  • Turn password_encryption GUC into an enum. This makes the parameter easier to extend, to support other password-based authentication protocols than MD5. (SCRAM is being worked on.) The GUC still accepts on/off as aliases for "md5" and "plain", although we may want to remove those once we actually add support for another password hash type. Michael Paquier, reviewed by David Steele, with some further edits by me. Discussion: <CAB7nPqSMXU35g=W9X74HVeQp0uvgJxvYOuA4A-A3M+0wfEBv-w@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/babe05bc2b781eb3eb84a18d7010d08277e2e399
  • Don't bother to lock bufmgr partitions in pg_buffercache. That makes the view a lot less disruptive to use on a production system. Without the locks, you don't get a consistent snapshot across all buffers, but that's OK. It wasn't a very useful guarantee in practice. Ivan Kartyshov, reviewed by Tomas Vondra and Robert Haas. Discusssion: <f9d6cab2-73a7-7a84-55a8-07dcb8516ae5@postgrespro.ru> http://git.postgresql.org/pg/commitdiff/6e654546fb61f62cc982d0c8f62241b3b30e7ef8

Robert Haas pushed:

Stephen Frost pushed:

  • Remove superuser checks in pgstattuple. Now that we track initial privileges on extension objects and changes to those permissions, we can drop the superuser() checks from the various functions which are part of the pgstattuple extension and rely on the GRANT system to control access to those functions. Since a pg_upgrade will preserve the version of the extension which existed prior to the upgrade, we can't simply modify the existing functions but instead need to create new functions which remove the checks and update the SQL-level functions to use the new functions (and to REVOKE EXECUTE rights on those functions from PUBLIC). Thanks to Tom and Andres for adding support for extensions to follow update paths (see: 40b449a), allowing this patch to be much smaller since no new base version script needed to be included. Approach suggested by Noah. Reviewed by Michael Paquier. http://git.postgresql.org/pg/commitdiff/fd321a1dfd64d30bf1652ea6b39b654304f68ae4

Magnus Hagander pushed:

Correctifs en attente

Takayuki Tsunakawa sent in two revisions of a patch to implement huge_pages on Windows.

Michaël Paquier and Heikki Linnakangas traded patch flocks to implement SCRAM password auth.

Pavel Stěhule sent in another revision of a patch to add a \setfileref command to psql.

Pavel Stěhule sent in another revision of a patch to add xmltable().

Vitaly Burovoy sent in a patch to detect supported SET parameters when pg_restore is run.

Vinayak Pokale support prepared transactions with foreign servers.

Peter Eisentraut and Michaël Paquier traded patches to fix CRC check handling in get_controlfile.

Stephen Frost and Jeevan Chalke traded patches to add support for restrictive RLS policies.

Jesper Pedersen and Peter Eisentraut traded patches to add hash index support to pageinspect.

Kyotaro HORIGUCHI and Michaël Paquier traded patches to track LSN on lightly loaded systems to reduce the number of checkpoints.

Tomas Vondra sent in another revision of a patch to add two slab-like memory allocators.

Fabien COELHO sent in three more revisions of a patch to add more operators and functions to pgbench.

Fabien COELHO sent in another revision of a patch to enable storing select results into variables.

Rajkumar Raghuwanshi sent in a patch to add test cases for partition-wise joins on declarative partitions.

Daniel Gustafsson sent in a patch to make flex/bison checks stricter in Git trees.

Mark Dilger sent in a patch to stop casting away const in comparators.

Mark Dilger sent in a patch to stop casting away const in regex.

Mark Dilger sent in a patch to avoid casting away const when using the FileName typedef.

Takayuki Tsunakawa sent in a patch to allow SIGQUIT to prompt a fast shutdown of the stats collector without leaving partly-written stats files behind.

Mark Dilger sent in a WIP patch to const-ify xlog_outdesc() and rm_redo_error_callback().

David Steele and Michaël Paquier traded patches to exclude additional directories in pg_basebackup.

Amit Langote sent in three more revisions of a patch to implement declarative partitions.

Michaël Paquier sent in four more revisions of a patch to track wait event for latches.

Victor Wagner sent in two more revisions of a patch to implement failover at the libpq connect level.

Mark Dilger sent in a patch to tidy up some handling of pointers in rangetypes_spgist.c, tsgistidx.c, and tsrank.c.

David Cramer and Heikki Linnakangas traded patches to add support for multi-dimensional arrays to PL/PythonU.

Mithun Cy sent in another revision of a patch to cache hash index meta pages.

Emre Hasegeli sent in another revision of a patch to fix floating point comparison inconsistencies in the geometric types.

Ivan Kartyshov sent in another revision of a patch to make pg_buffercache more efficient with large shared memory.

Etsuro Fujita sent in two more revisions of a patch to push down more full joins to the PostgreSQL FDW.

David Fetter and Thomas Munro traded patches to add a hook and corresponding GUCs to disable UPDATEs and DELETEs that lack a WHERE clause.

Michaël Paquier sent in another revision of a patch to rename pg_xlog to pg_wal and rename pg_clog to pg_transaction.

Heikki Linnakangas sent in two more revisions of a patch to change the way use memory for larger read buffers in logtape.c rather than pre-reading tuples into SortTuple slots during merge.

Magnus Hagander sent in two more revisions of a patch to enable pg_basebackup to stream xlog to tar.

Kyotaro HORIGUCHI sent in another revision of a patch to fix wal level minimal.

Ashutosh Bapat sent in another revision of a patch to fix an issue where altering foreign table was not invalidating prepare statement execution plans that it should have.

Jeevan Chalke sent in another revision of a patch to enable pushing aggregates to a FDW.

Aleksander Alekseev sent in a patch to rename md5Salt to pwsalt.

Corey Huinker sent in a PoC patch to make a copy() set-returning function.

Stephen Frost sent in another revision of a patch to fix some infelicities between COPY and RLS.

Andres Freund sent in another revision of a patch to add a macro-customizable hashtable for bitmapscan and aggregation performance.

Daniel Vérité sent in another revision of a patch to fix some pg_dump / copy bugs with long lines.

Dmitry Dolgov sent in another revision of a patch to add a generic type subscription.

Emre Hasegeli sent in a patch to add <@, @>, <<@, and @>> operator symbols for the inet datatype to replace <<=, >>=, <<, and >>.

Christoph Berg sent in a patch to ensure that the default log_line prefix is not empty.

par N Bougain le lundi 10 octobre 2016 à 21h49

lundi 3 octobre 2016

Actualités PostgreSQL.fr

Sortie de PostgreSQL 9.6

29 septembre 2016 : le PostgreSQL Global Development Group a publié aujourd'hui PostgreSQL 9.6, dernière version du système libre de gestion de bases de données SQL de référence. Cette version va permettre aux utilisateurs de réaliser à la fois une expansion interne (scale up) et externe (scale out, répartition de la charge) de leurs bases de données haute-performance. Les nouvelles fonctionnalités incluent notamment le parallélisme des requêtes, des améliorations sur la réplication synchrone, la recherche par phrase, ainsi que des améliorations sur la performance et la facilité d'utilisation.

Expansion interne avec le parallélisme des requêtes

La version 9.6 ajoute le support du parallélisme pour certaines opérations dans les requêtes. Ce parallélisme active l'utilisation de tout ou partie des cœurs de processeur d'un serveur pour retourner les résultats plus rapidement. Sur cette version, le parallélisme concerne le parcours séquentiel (des tables), les agrégats et les jointures. En fonction des caractéristiques et du nombre de cœurs, le parallélisme permet d'espérer des gains jusqu'à un facteur 32 sur des requêtes traitant des volumes importants de données.

"J'ai migré l'intégralité de notre plateforme génomique - soit un héritage de 25 milliards de lignes MySQL - vers une seule base de données Postgres, en utilisant les possibilités de compression du type JSONB, les excellentes méthodes d'indexation GIN, BRIN et B-tree. Aujourd'hui, avec la version 9.6, j'attends de pouvoir exploiter le parallélisme des requêtes pour améliorer encore plus les performances des requêtes sur les tables volumineuses" indique Mike Sofen, Chief Database Architect chez Synthetic Genomics.

Répartition de charge avec la réplication synchrone et postgres_fdw

Deux nouvelles options ont été ajoutées à la réplication synchrone de PostgreSQL. Elles permettent de maintenir des lectures cohérentes de données sur les nœuds d'un cluster de bases de données. En premier lieu, il est désormais possible de configurer des groupes de réplication synchrone. En second lieu, le mode "remote_apply" (application distante) crée une vue plus cohérente des données sur les différents nœuds. Ces fonctionnalités utilisent la réplication interne pour maintenir un groupe de nœuds "identiques" afin d'équilibrer la charge de lecture (load-balancing).

Le pilote de fédération de données entre bases PostgreSQL, postgres_fdw, dispose de nouvelles fonctionnalités d'exécution du travail sur le serveur distant. En "t;externalisant" les tris, les jointures et les mises à jour par lot, la charge peut être distribuée sur plusieurs serveurs PostgreSQL. Ces possibilités seront bientôt ajoutées aux autres pilotes FDW.

"Avec les possibilités d'externaliser les jointures, les mises à jour et les suppressions, les Foreign Data Wrappers offrent maintenant une solution complète de partage de données entre PostgreSQL et les autres bases de données. Par exemple, PostgreSQL peut être utilisé pour gérer en entrée des données qui vont vers des bases de données différentes" indique Julyanto Sutandang, Director of Business Solutions chez Equnix.

Meilleure recherche de texte avec des phrases

La fonctionnalité de recherche plein texte de PostgreSQL supporte désormais les recherche par phrases. Cela permet une recherche de phrases exactes ou de mots proches les uns des autres, en utilisant l'indexation GIN. Additionné aux nouvelles fonctionnalités d'optimisation de la recherche textuelle, cet apport fait de PostgreSQL un outil de choix pour la "recherche hybride" mêlant relationnel, JSON et recherche plein texte.

Plus agréable, plus rapide et plus facile à utiliser

Grâce au retours d'expérience et aux tests effectués par les utilisateurs de PostgreSQL disposant de bases de données de production à fort volume, le projet a pu améliorer de nombreux aspects des performances et de la maniabilité dans cette version. La réplication, les agrégats, l'indexation, les tris et les procédures stockées sont plus efficaces, et PostgreSQL utilise désormais mieux les ressources lorsqu'il est installé sur des noyaux Linux récents. Le surcoût d’administration des tables volumineuses et des charges complexes a été réduit, notamment par des améliorations du VACUUM.

Détail des fonctionnalités

Plus d'informations sur les fonctionnalités ci-dessus et les autres dans les liens suivants :

Téléchargement

Documentation

La documentation au format HTML et les pages de manuel sont installées avec PostgreSQL. La documentation en ligne, exhaustive et interactive, peut être parcourue, interrogée et commentée librement.

Licence

PostgreSQL utilise la licence PostgreSQL, une licence permissive de type BSD. Cette licence certifiée par l'OSI est largement appréciée pour sa flexibilité et sa compatibilité avec le monde des affaires, puisqu'elle ne restreint pas l'utilisation de PostgreSQL dans les applications propriétaires ou commerciales. Associée à un support proposé par de multiples sociétés et une propriété publique du code, sa licence rend PostgreSQL très populaire parmi les revendeurs souhaitant embarquer une base de données dans leurs produits sans avoir à se soucier des prix de licence, des verrous commerciaux ou modifications des termes de licence.

Contacts

Pages Web

Contact presse

France et pays francophones
Stéphane Schildknecht
fr at postgresql dot org
+33 (0) 617 11 37 42

Pour les contacts d'autres régions, consulter la liste des contacts internationaux.

Informations concernant les sociétés citées et texte intégral des citations

"En tant qu'architecte en chef de bases de données chez Synthetic Genomics, j'ai été réellement impressionné par la puissance, la flexibilité et la richesse des interfaces de programmation de Postgres 9.5. J'ai migré l'intégralité de notre plateforme génomique — soit 25 milliards de lignes issues de MySQL — vers une seule base de données Postgres, en utilisant les possibilités de compression du type JSONB, les excellentes méthodes d'indexation GIN, BRIN et B-tree, ainsi que l'élégant langage de programmation de fonctions stockées PL/pgSQL. Aujourd'hui, avec la version 9.6, j'attends de pouvoir exploiter le parallélisme des requêtes pour améliorer encore plus les performances des requêtes sur les tables volumineuses. Postgres 9.5 a prouvé par lui-même qu'il est complètement stable, remarquablement performant, et selon moi, le meilleur moteur à ce jour de stockage de données hybride relationnel-document sur le marché." indique Mike Sofen, Chief Database Architect chez Synthetic Genomics.

Synthetic Genomics, Inc. (SGI) est l'un des leaders mondiaux dans les domaines de la biologie de synthèse et la génomique de synthèse. SGI se fonde sur l'utilisation de la propriété intellectuelle dans cette discipline qui évolue rapidement, pour concevoir et réaliser des systèmes biologiques qui permettent de résoudre les défis de développement durable à l'échelle de la planète. SGI fournit des solutions dans les domaines de la recherche génomique, de la bio-production et des produits appliqués. La recherche est focalisée sur les solutions génomiques pour tout organisme de recherche et développement, académique ou commercial. Les produits et services commerciaux incluent l'instrumentation, les réactifs, les services de synthèse d'ADN et les logiciels et services en bio-informatique. Contact : Synthetic Genomics, media@syntheticgenomics.com (anglais)

"Avec les possibilités d'externaliser les jointures, les mises à jour et les suppressions, les Foreign Data Wrappers offrent maintenant une solution complète de partage de données entre PostgreSQL et les autres bases de données. Par exemple, PostgreSQL peut être utilisé pour gérer en entrée des données qui vont vers des bases de données différentes" indique Julyanto Sutandang, Director of Business Solutions chez Equnix.

Equnix est fournisseur de solutions informatiques pour les entreprises. Cette société met en place des solutions alternatives complètes, basées sur la recherche, le développement et du code libre. Equnix fournit du service autour de PostgreSQL. Contact en anglais ou en indonésien : Info@equnix.co.id ou téléphoner à Johan D. au +6221-22866662.

Support Professionnel

PostgreSQL bénéficie du support de nombreuses sociétés, qui financent des développeurs, fournissent l'hébergement ou un support financier. Les plus fervents supporters sont listés sur la page des mécènes du développement.

Il existe également une très grande communauté de sociétés fournissant du support PostgreSQL, des consultants indépendants aux sociétés multinationales.

Les dons sont acceptés avec plaisir.

Vous pouvez également acheter des produits dérivés de qualité sur la Boutique Zazzle PostgreSQL.

par SAS le lundi 3 octobre 2016 à 05h33

jeudi 29 septembre 2016

Sébastien Lardière

PostgreSQL 9.6.0

La version 9.6.0 de PostgreSQL est publiée aujourd'hui. La fonctionnalité la plus notable de cette version majeure est la parallélisation des requêtes. De nombreuses autres fonctionnalités permettent de travailler sur des volumes de données toujours plus important. Quelques informations sur le site de Loxodata :... Lire PostgreSQL 9.6.0

par Sébastien Lardière le jeudi 29 septembre 2016 à 14h16

mardi 27 septembre 2016

Actualités PostgreSQL.fr

Nouvelles hebdomadaires de PostgreSQL - 25 septembre 2016

Les nouveautés des produits dérivés

Offres d'emplois autour de PostgreSQL en septembre

PostgreSQL Local

PostgreSQL dans les média

PostgreSQL Weekly News / les nouvelles hebdomadaires vous sont offertes cette semaine par David Fetter. Traduction par l'équipe PostgreSQLFr sous licence CC BY-NC-SA. La version originale se trouve à l'adresse suivante : http://www.postgresql.org/message-id/20160925203116.GB17775@fetter.org

Proposez vos articles ou annonces avant dimanche 15:00 (heure du Pacifique). Merci de les envoyer en anglais à david (a) fetter.org, en allemand à pwn (a) pgug.de, en italien à pwn (a) itpug.org et en espagnol à pwn (a) arpug.com.ar.

Correctifs appliqués

Robert Haas pushed:

Heikki Linnakangas pushed:

Peter Eisentraut pushed:

Tom Lane pushed:

  • Be sure to rewind the tuplestore read pointer in non-leader CTEScan nodes. ExecInitCteScan supposed that it didn't have to do anything to the extra tuplestore read pointer it gets from tuplestore_alloc_read_pointer. However, it needs this read pointer to be positioned at the start of the tuplestore, while tuplestore_alloc_read_pointer is actually defined as cloning the current position of read pointer 0. In normal situations that accidentally works because we initialize the whole plan tree at once, before anything gets read. But it fails in an EvalPlanQual recheck, as illustrated in bug #14328 from Dima Pavlov. To fix, just forcibly rewind the pointer after tuplestore_alloc_read_pointer. The cost of doing so is negligible unless the tuplestore is already in TSS_READFILE state, which wouldn't happen in normal cases. We could consider altering tuplestore's API to make that case cheaper, but that would make for a more invasive back-patch and it doesn't seem worth it. This has been broken probably for as long as we've had CTEs, so back-patch to all supported branches. Discussion: <32468.1474548308@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/96dd77d349424f270d129f8f40e75f762ddcca7d
  • Remove nearly-unused SizeOfIptrData macro. Past refactorings have removed all but one reference to SizeOfIptrData (and that one place was in a pretty noncritical spot). Since nobody's complained, it seems probable that there are no supported compilers that don't think sizeof(ItemPointerData) is 6. If there are, we're wasting MAXALIGN per heap tuple anyway, so it's rather silly to worry about whether we can shave space in places like WAL records. Pavan Deolasee Discussion: <CABOikdOOawDda4hwLOT6zdA6MFfPLu3Z2YBZkX0JdayNS6JOeQ@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/8023b5827fbada6815ce269db4f3373ac77ec7c3
  • Avoid using PostmasterRandom() for DSM control segment ID. Commits 470d886c3 et al intended to fix the problem that the postmaster selected the same "random" DSM control segment ID on every start. But using PostmasterRandom() for that destroys the intended property that the delay between random_start_time and random_stop_time will be unpredictable. (Said delay is probably already more predictable than we could wish, but that doesn't mean that reducing it by a couple orders of magnitude is OK.) Revert the previous patch and add a comment warning against misuse of PostmasterRandom. Fix the original problem by calling srandom() early in PostmasterMain, using a low-security seed that will later be overwritten by PostmasterRandom. Discussion: <20789.1474390434@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/49a91b88e6c4afb840745c78942dd99ce125a6d6
  • Don't trust CreateFileMapping() to clear the error code on success. We must test GetLastError() even when CreateFileMapping() returns a non-null handle. If that value were left over from some previous system call, we might be fooled into thinking the segment already existed. Experimentation on Windows 7 suggests that CreateFileMapping() clears the error code on success, but it is not documented to do so, so let's not rely on that happening in all Windows releases. Amit Kapila Discussion: <20811.1474390987@sss.pgh.pa.us> http://git.postgresql.org/pg/commitdiff/8e6b4ee21f486e6800aaa6955ff3d98e1a521b49
  • Remove useless code. Apparent copy-and-pasteo in standby_desc_invalidations() had two entries for msg->id == SHAREDINVALRELMAP_ID. Aleksander Alekseev Discussion: <20160923090814.GB1238@e733> http://git.postgresql.org/pg/commitdiff/959ea7fa76120449efa5d85ae351abd1d6e5480c
  • Fix incorrect logic for excluding range constructor functions in pg_dump. Faulty AND/OR nesting in the WHERE clause of getFuncs' SQL query led to dumping range constructor functions if they are part of an extension and we're in binary-upgrade mode. Actually, we don't want to dump them separately even then, since CREATE TYPE AS RANGE will create the range's constructor functions regardless. Per report from Andrew Dunstan. It looks like this mistake was introduced by me, in commit b985d4877, in perhaps-overzealous refactoring to reduce code duplication. I'm suitably embarrassed. Report: <34854939-02d7-f591-5677-ce2994104599@dunslane.net> http://git.postgresql.org/pg/commitdiff/12f6eadffd075e39570a0170f6791de0b96ddfde
  • Doc: fix examples of # operators so they actually work. These worked as-is until around 7.0, but fail in newer versions because there are more operators named "#". Besides it's a bit inconsistent that only two of the examples on this page lack type names on their constants. Report: <20160923081530.1517.75670@wrigleys.postgresql.org> http://git.postgresql.org/pg/commitdiff/5a7bae0699657315b487896810a30bd7425f6a08
  • Install TAP test infrastructure so it's available for extension testing. When configured with --enable-tap-tests, "make install" will now install the Perl support files for TAP testing where PGXS will find them. This allows extensions to rely on $(prove_check) even when being built out-of-tree. Back-patch to 9.4 where we first started to support TAP testing, to reduce the number of cases extension makefiles need to consider. Craig Ringer Discussion: <CAMsr+YFXv+2qne6xJW7z_25mYBtktRX5rpkrgrb+DRgQ_FxgHQ@mail.gmail.com> http://git.postgresql.org/pg/commitdiff/c3a0818460a8f4a5e1a9109b035ac30e84b7e06f
  • Do a final round of updates on the 9.6 release notes. Set release date, document a few recent commits, do one last pass of copy-editing. http://git.postgresql.org/pg/commitdiff/98c2d3332b30006ff71add99bc9d619c9457e71f

Bruce Momjian pushed:

Correctifs en attente

David Fetter sent in another revision of a patch to add a contrib hook and associated GUCs for disallowing UPDATE and/or DELETE when no WHERE clause is present.

Vladimir Gordiychuk and Craig Ringer traded patches to permit pausing replication.

Thomas Munro sent in a patch to check the return code for postmaster death.

Takayuki Tsunakawa sent in a patch to make large shared_buffers settings effective on Windows.

Ashutosh Bapat sent in a patch to cache the result of code in add_paths_to_joinrel() which computes the set of target relations that should overlap parameterization of any proposed join path rather than re-computing it each time it's used in a query.

Daniel Vérité sent in another revision of a patch to enable psql variable hooks to return a boolean indicating whether a change is allowed.

Mithun Cy sent in another revision of a patch to add some tests to cover hash indexes.

Craig Ringer and Robert Haas traded patches to add a txid_status(bigint) to get status of an xact.

Jesper Pedersen sent in four more revisions of a patch to add hash index support to pageinspect.

Heikki Linnakangas and Fabien COELHO traded patches to refactor the pgbench meta commands into a state machine.

Mark Dilger sent in a patch to fix a place that shouldn't have cast away const.

Michaël Paquier sent in a patch to add suitable OOM warnings to palloc_extended() and pg_malloc_internal().

Haribabu Kommi sent in another revision of a patch to enable tracking the number of commands executed grouped by command tag.

Ashutosh Bapat sent in two more revisions of a patch to enable optimizations of partition-wise join for join between (declaratively) partitioned tables.

Michaël Paquier sent in three more revisions of a patch to track wait events for latches.

Amit Kapila sent in another revision of a patch to implement concurrent hash indexes.

David Cramer and Pavel Stěhule traded patches to add PL/PythonU support for multidimensional arrays.

Pavel Stěhule and Craig Ringer traded patches to add xmltable().

Anastasia Lubennikova sent in another revision of a patch to add covering & unique indexes.

Heikki Linnakangas sent in a patch to refactor the LogicalTapeSet/LogicalTape interface.

Ashutosh Bapat sent in a patch to refactor index creation so it plays better with declaratively partitioned tables.

Julian Markwort sent in a patch to allow specifying a 'custom' pgpassfile within the connection string along the lines of the existing parameters sslkey and sslcert.

Michaël Paquier sent in another revision of a patch to move sync routines of initdb into src/common, issue fsync more carefully in pg_receivexlog and pg_basebackup [-X] stream, add --nosync option to pg_basebackup, and switch pg_basebackup commands in Postgres.pm to use --no-sync.

Peter Eisentraut sent in another revision of a patch to change pg_dump by separate table and sequence data object types and fixing pg_upgrade to use this facility from pg_dump.

David Steele sent in another revision of a patch to exclude additional directories in pg_basebackup.

Robert Haas sent in a WIP patch to enable asynchronous execution.

Amit Kapila sent in another revision of a patch to do SetLastError(0) before calling CreateFileMapping.

Thomas Munro sent in a patch to add psql tab completions for LOCK TABLE ... IN ACCESS|ROW|SHARE.

Thomas Munro sent in a patch to refactor StartupXLOG.

Masahiko Sawada sent in another revision of a patch to add quorum commit.

Amit Kapila sent in another revision of a patch to add WAL for hash indexes.

par N Bougain le mardi 27 septembre 2016 à 21h34

samedi 13 août 2016

Adrien Nayrat

PostgreSQL : Retarder la vérification des contraintes

Retarder la vérification des contraintes

Remarque : Cet article a été rédigé durant le cadre de mon activité chez Dalibo.

Postgres respecte le modèle ACID, ainsi il garantie la cohérence de la base : une transaction amène la base d’un état stable à un autre.

Les données dans les différentes tables ne sont pas indépendantes mais obéissent à des règles sémantiques mises en place au moment de la conception du modèle conceptuel des données. Les contraintes d’intégrité ont pour principal objectif de garantir la cohérence des données entre elles, et donc de veiller à ce qu’elles respectent ces règles sémantiques. Si une insertion, une mise à jour ou une suppression viole ces règles, l’opération est purement et simplement annulée.

Le moteur effectue la vérification des contraintes à chaque modification (lorsque des contraintes ont été définies). Il est également possible de retarder la vérification des contraintes à la fin de la transaction, au moment du commit. Ainsi, les vérifications ne seront produites que sur les changements effectifs entre les opérations de delete, update et insert de la transaction.

Exemple :

CREATE TABLE t1 (c1 INT PRIMARY KEY, c2 text);
 CREATE TABLE t2 (c1 INT REFERENCES t1(c1), c2 text);

INSERT INTO t1  VALUES(1,'a');
 INSERT INTO t1  VALUES(2,'b');

INSERT INTO t2  VALUES(3,'a');
 ERROR:  INSERT OR UPDATE ON TABLE "t2" violates FOREIGN KEY CONSTRAINT "t2_c1_fkey"
 DETAIL:  KEY (c1)=(3) IS NOT present IN TABLE "t1".

La ligne insérée dans t2 doit respecter la contrainte d’intégrité référentielle, la table t1 ne contient aucune ligne où c1=3. Insérons une ligne correcte :

INSERT INTO t2  VALUES(1,'a');

SELECT * FROM t1;
 c1 | c2
 ----+----
 1 | a
 2 | b
 (2 ROWS)

SELECT * FROM t2;
 c1 | c2
 ----+----
 1 | a
 (1 ROW)

Que se passe t-il si on souhaite modifier la clé primaire de t1?

BEGIN;
 UPDATE t1 SET c1=3 WHERE c1=1;
 ERROR:  UPDATE OR DELETE ON TABLE "t1" violates FOREIGN KEY CONSTRAINT "t2_c1_fkey" ON TABLE "t2"
 DETAIL:  KEY (c1)=(1) IS still referenced FROM TABLE "t2".

La vérification de la contrainte se fait lors de l’UPDATE et déclenche une erreur. Il est possible de demander au moteur d’effectuer la vérification des contraintes à la fin de la transaction avec l’ordre SET CONSTRAINTS ALL DEFERRED;. A noter également que le mot clef ALL peut-être remplacé par le nom d’une contrainte (si elle est nommée et est DEFERRABLE)

BEGIN;
 SET CONSTRAINTS ALL DEFERRED;
 UPDATE t1 SET c1=3 WHERE c1=1;
 ERROR:  UPDATE OR DELETE ON TABLE "t1" violates FOREIGN KEY CONSTRAINT "t2_c1_fkey" ON TABLE "t2"
 DETAIL:  KEY (c1)=(1) IS still referenced FROM TABLE "t2".

Ça ne fonctionne toujours pas, en effet il faut préciser que que l’application de la contrainte peut être retardée avec le mot clé DEFERRABLE

ALTER TABLE t2 ALTER CONSTRAINT t2_c1_fkey DEFERRABLE;

BEGIN;
 SET CONSTRAINTS ALL DEFERRED;
 UPDATE t1 SET c1=3 WHERE c1=1;
 UPDATE t2 SET c1=3 WHERE c1=1;
 commit;

SELECT * FROM t1;
 c1 | c2
 ----+----
 2 | b
 3 | a
 (2 ROWS)

SELECT * FROM t2;
 c1 | c2
 ----+----
 3 | a
 (1 ROW)

Dans ce cas le moteur accepte de faire la vérification en fin de transaction.

Autre intérêt, si une ligne est effacée et réinsérée dans la même transaction, les vérifications sur cette ligne ne sont pas exécutées (car inutiles).

Exemple, on vide les tables puis on insère 1 million de lignes.

TRUNCATE t1 cascade;
 NOTICE:  TRUNCATE cascades TO TABLE "t2"
 TRUNCATE TABLE

EXPLAIN analyse INSERT INTO T1 (c1,c2) (SELECT  *,md5(i::text) FROM (SELECT * FROM generate_series(1,1000000)) i);

Ensuite on insère 100 000 lignes, puis on les supprime pour les réinsérer à nouveau (sans indiquer au moteur de différer la vérification de contraintes).

BEGIN;

EXPLAIN analyse  INSERT INTO T2 (c1,c2) (SELECT  *,md5(i::text) FROM (SELECT * FROM generate_series(1,1000000)) i);

QUERY PLAN
 ------------------------------------------------------------------------------------------------------------------------------------
 INSERT ON t2  (cost=0.00..17.50 ROWS=1000 width=36) (actual TIME=3451.308..3451.308 ROWS=0 loops=1)
 -&gt;  FUNCTION Scan ON generate_series  (cost=0.00..17.50 ROWS=1000 width=36) (actual TIME=170.218..1882.406 ROWS=1000000 loops=1)
 Planning TIME: 0.054 ms
 TRIGGER FOR CONSTRAINT t2_c1_fkey: TIME=16097.543 calls=1000000
 Execution TIME: 19654.595 ms
 (5 ROWS)

TIME: 19654.971 ms

DELETE FROM t2 WHERE c1 &lt;= 1000000;
 DELETE 1000000
 TIME: 2088.318 ms

EXPLAIN analyse  INSERT INTO T2 (c1,c2) (SELECT  *,md5(i::text) FROM (SELECT * FROM generate_series(1,1000000)) i);
 QUERY PLAN
 ------------------------------------------------------------------------------------------------------------------------------------
 INSERT ON t2  (cost=0.00..17.50 ROWS=1000 width=36) (actual TIME=3859.265..3859.265 ROWS=0 loops=1)
 -&gt;  FUNCTION Scan ON generate_series  (cost=0.00..17.50 ROWS=1000 width=36) (actual TIME=169.826..1845.421 ROWS=1000000 loops=1)
 Planning TIME: 0.054 ms
 TRIGGER FOR CONSTRAINT t2_c1_fkey: TIME=14600.258 calls=1000000
 Execution TIME: 18558.108 ms
 (5 ROWS)

TIME: 18558.386 ms
 commit;
 COMMIT
 TIME: 8.563 ms

Le moteur va effectuer les vérifications à chaque insertion (environ 18 secondes à chaque insertion).

Effectuons la même opération en retardant la vérification des contraintes :

BEGIN;
 BEGIN
 TIME: 0.130 ms

SET CONSTRAINTS ALL deferred ;
 SET CONSTRAINTS

EXPLAIN analyse  INSERT INTO T2 (c1,c2) (SELECT  *,md5(i::text) FROM (SELECT * FROM generate_series(1,1000000)) i);

QUERY PLAN
 ------------------------------------------------------------------------------------------------------------------------------------
 INSERT ON t2  (cost=0.00..17.50 ROWS=1000 width=36) (actual TIME=3241.172..3241.172 ROWS=0 loops=1)
 -&gt;  FUNCTION Scan ON generate_series  (cost=0.00..17.50 ROWS=1000 width=36) (actual TIME=169.770..1831.893 ROWS=1000000 loops=1)
 Planning TIME: 0.096 ms
 Execution TIME: 3269.624 ms
 (4 ROWS)

TIME: 3270.018 ms

DELETE FROM t2 WHERE c1 &lt;= 1000000;
 DELETE 2000000
 TIME: 2932.070 ms

EXPLAIN analyse  INSERT INTO T2 (c1,c2) (SELECT  *,md5(i::text) FROM (SELECT * FROM generate_series(1,1000000)) i);
 QUERY PLAN
 ------------------------------------------------------------------------------------------------------------------------------------
 INSERT ON t2  (cost=0.00..17.50 ROWS=1000 width=36) (actual TIME=3181.294..3181.294 ROWS=0 loops=1)
 -&gt;  FUNCTION Scan ON generate_series  (cost=0.00..17.50 ROWS=1000 width=36) (actual TIME=170.137..1811.889 ROWS=1000000 loops=1)
 Planning TIME: 0.055 ms
 Execution TIME: 3209.712 ms
 (4 ROWS)

TIME: 3210.067 ms

commit;
 COMMIT
 TIME: 16630.155 ms

Les insertions sont plus rapides, en revanche le commit est plus long car le moteur effectue la vérification des contraintes. Au final, le moteur effectue une seule vérification à la fin de la transaction (commit). L’opération est donc plus rapide.

Il est possible de créer la contrainte avec un autre attribut DEFERRABLE INITIALLY DEFERRED qui permet de s’affranchir du SET CONSTRAINTS ALL DEFERRED. A noter également que le mot clef ALL peut-être remplacé par le nom d’une contrainte (si elle est nommée et est DEFERRABLE)

Si les enregistrements modifiés ne respectent pas les contraintes, la transaction est annulée au moment du commit :

SELECT * FROM t1;
 c1 | c2
 ----+----
 1 | un
 (1 ROW)

BEGIN;
 SET constraints ALL deferred ;
 SET CONSTRAINTS
 anayrat=# INSERT INTO t2 VALUES ('2','un');
 INSERT 0 1
 anayrat=# commit;
 ERROR:  INSERT OR UPDATE ON TABLE "t2" violates FOREIGN KEY CONSTRAINT "t2_c1_fkey"
 DETAIL:  KEY (c1)=(2) IS NOT present IN TABLE "t1".

D’autre part, la solution de désactivation des contraintes peut effectivement poser des problèmes si on souhaite les réactiver et que les données ne le permettent pas (contrainte effectivement rompue), cette solution reste possible, au sein d’une transaction, toutefois cela provoque un verrouillage exclusif sur la table modifiée pendant toute la transaction ce qui peut poser de sérieux problèmes de performance.

Il est également possible de déclarer la contrainte en NOT VALID. La création de la contrainte sera quasi immédiate, les données actuellement présentes ne seront pas validées. Cependant, toutes les données insérées ou mises à jour par la suite seront validées vis à vis de ces contraintes.

Ensuite on peut demander au moteur de faire la vérification des contraintes pour l’intégralité des enregistrements avec l’ordre VALIDATE CONSTRAINT. Cet ordre entraîne un verrou exclusif sur la table. A partir de la version 9.4 le verrou est plus léger : SHARE UPDATE EXCLUSIVE sur la table modifiée. Si la contrainte est une clé étrangère, le verrou est de type ROW SHARE sur la table référente.

ALTER TABLE t2 DROP CONSTRAINT t2_c1_fkey ;
 ALTER TABLE
 SELECT * FROM t1;
 c1 | c2
 ----+----
 1 | un
 (1 ROW)

SELECT * FROM t2;
 c1 | c2
 ----+----
 (0 ROWS)

INSERT INTO t2 VALUES (2,'deux');
 INSERT 0 1
 SELECT * FROM t2;
 c1 |  c2
 ----+------
 2 | deux
 (1 ROW)

ALTER TABLE t2 ADD CONSTRAINT t2_c1_fkey FOREIGN KEY (c1) REFERENCES t1(c1) NOT VALID;
 ALTER TABLE

La table t2 contient un enregistrement ne respectant pas la contrainte t2_c1_fkey. Aucune erreur n’est remontée car la vérification se fait seulement pour les nouvelles modifications :

INSERT INTO t2 VALUES (3,'trois');
 ERROR:  INSERT OR UPDATE ON TABLE "t2" violates FOREIGN KEY CONSTRAINT "t2_c1_fkey"
 DETAIL:  KEY (c1)=(3) IS NOT present IN TABLE "t1".

De même, une erreur est bien remontée lors de la vérification de contrainte :

ALTER TABLE t2 VALIDATE CONSTRAINT t2_c1_fkey ;
 ERROR:  INSERT OR UPDATE ON TABLE "t2" violates FOREIGN KEY CONSTRAINT "t2_c1_fkey"
 DETAIL:  KEY (c1)=(2) IS NOT present IN TABLE "t1".

En supprimant l’enregistrement posant problème, les contraintes peuvent être validées :

DELETE FROM t2 WHERE t2.c1 NOT IN (SELECT c1 FROM t1);
 DELETE 1
 ALTER TABLE t2 VALIDATE CONSTRAINT t2_c1_fkey ;
 ALTER TABLE

Partager

par Adrien Nayrat le samedi 13 août 2016 à 20h42

vendredi 20 mai 2016

Guillaume Lelarge

Quelques nouvelles sur les traductions du manuel

J'ai passé beaucoup de temps ces derniers temps sur la traduction du manuel de PostgreSQL.

  • mise à jour pour les dernières versions mineures
  • corrections permettant de générer les PDF déjà disponibles (9.1, 9.2, 9.3 et 9.4) mais aussi le PDF de la 9.5
  • merge pour la traduction de la 9.6

Elles sont toutes disponibles sur le site docs.postgresql.fr.

De ce fait, la traduction du manuel de la 9.6 peut commencer. Pour les intéressés, c'est par là : https://github.com/gleu/pgdocs_fr/wiki/Translation-9.6

N'hésitez pas à m'envoyer toute question si vous êtes intéressé pour participer.

par Guillaume Lelarge le vendredi 20 mai 2016 à 20h35

jeudi 21 avril 2016

Adrien Nayrat

Index BRIN – Performances

La version 9.5 de PostgreSQL sortie en Janvier 2016 propose un nouveau type d’index : les Index BRIN pour Bloc Range INdex. Ces derniers sont recommandés pour les tables volumineuses et corrélées avec leur emplacement. J’ai décidé de consacrer une série d’article sur ces index :

Pour information, je serai présent au PGDay France à Lille le mardi 31 mai pour présenter cet index. Il y aura également plein d’autres conférences intéressantes!

Cet article est la dernier de la série, il sera consacré aux performances (maintenance, lecture, insertion…)

Performances

Les articles précédents ont abordé le fonctionnement et les spécificités des index BRIN. Cet article sera plutôt consacré aux performances. Les exemples précédents portaient sur de petites volumétries. Maintenant nous allons voir ce que peuvent apporter ces index sur une volumétrie plus importante.

Les tests ont été effectués sur un PC portable, les journaux de transaction, la table et les index sont stockés sur un disque mécanique. Les résultats seront différents suivant la matériel utilisé. Ces chiffres sont purement indicatifs et servent surtout à se donner un ordre d’idée.

Exemple

Dans un premier temps il est nécessaire de créer une table avec une volumétrie importante.

Par exemple : un système de mesure avec 100 sondes et une mesure toutes les secondes. Nous allons donc obtenir 100*365*24*3600 mesures => un peu plus de 3 milliards de lignes.

-- Utilisation d'une fonction pour générer du texte aléatoire.
-- Trouvée ici : http://stackoverflow.com/questions/3970795/how-do-you-create-a-random-string-in-postgresql

CREATE OR REPLACE FUNCTION random_string(LENGTH INTEGER) RETURNS text AS 
$$
DECLARE
  chars text[] := '{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z}';
  RESULT text := '';
  i INTEGER := 0;
BEGIN
  IF LENGTH &lt; 0 THEN
    raise exception 'Given length cannot be less than 0';
  END IF;
  FOR i IN 1..LENGTH loop
    RESULT := RESULT || chars[1+random()*(array_length(chars, 1)-1)];
  END loop;
  RETURN RESULT;
END;
$$ LANGUAGE plpgsql;
-- Creation de la table contenant les sondes 
CREATE TABLE probe (id serial PRIMARY KEY, name text);
INSERT INTO probe (name ) SELECT random_string(5) FROM generate_series(1,100);

CREATE TABLE data AS
WITH generation AS (
SELECT '2015-01-01'::TIMESTAMP + i * INTERVAL '1 second' AS date_metric,sonde::text,random() AS metric
FROM generate_series(0, 3600*24*365) i,
LATERAL (SELECT name FROM probe) sonde)
SELECT * FROM generation;

La table obtenue fait un peu plus de 150Go, elle ne rentre donc pas dans  la RAM de la machine hébergeant l’instance et encore moins dans la mémoire partagée de PostgreSQL.

Maintenance

On crée plein d’index pour comparer leur taille :

CREATE INDEX metro_btree_idx ON data USING btree (date_metric);
CREATE INDEX metro_brin_idx_8 ON data USING brin (date_metric) WITH (pages_per_range = 8);
CREATE INDEX metro_brin_idx_16 ON data USING brin (date_metric) WITH (pages_per_range = 16);
CREATE INDEX metro_brin_idx_32 ON data USING brin (date_metric) WITH (pages_per_range = 32);
CREATE INDEX metro_brin_idx_64 ON data USING brin (date_metric) WITH (pages_per_range = 64);
CREATE INDEX metro_brin_idx_128 ON data USING brin (date_metric);
CREATE INDEX metro_brin_idx_256 ON data USING brin (date_metric) WITH (pages_per_range = 256);

Voici les résultats obtenus pour la durée de création des index et leurs tailles :

size-largesize-large

La création de l’index a été 4 fois plus rapide pour les index BRIN. Il est possible que leur création aurait été plus rapide avec un stockage plus performant.

La taille des index est également frappante,  l’index b-tree fait 66 Go alors que l’index BRIN avec le pages_per_range par défaut fait seulement 5 Mo.

On peut tout suite constater le gain sur l’espace utilisé et la rapidité de création des index. Les opérations de maintenances (REINDEX) seront grandement facilitées.

Performances en lecture

Nous allons effectuer plusieurs tests, l’idée est d’essayer de mettre en valeur les différences de comportements entre les index BRIN et b-tree.

La requête utilisée sera tout simple :

EXPLAIN (ANALYSE,BUFFERS,VERBOSE) SELECT date_metric,sonde,metric FROM DATA WHERE date_metric = 'xxx';

pour obtenir un résultat avec peu de lignes  :

WHERE date_metric = '2015-05-01 00:00:00'::TIMESTAMP

Pour obtenir plus de résultat on prendra un intervalle avec :

WHERE date_metric BETWEEN 'xxx'::TIMESTAMP AND 'xxx'::TIMESTAMP;

Voici les résultats obtenus :

lignes BRIN Btree Gain
Durée Blocs lus Durée Blocs lus Durée Volume données lues
100 24ms 697 0.06ms 7 Btree (x400) Btree (x100)
267 millions 170s 13Go 228s 18Go BRIN (x1.3) BRIN (x1.4)
777 millions 8min 38Go 11min 54Go BRIN (x1.37) BRIN (x1.4)
1.3 milliard 13min 63Go 32min (seqscan)
18min
153 Go (seqscan)
90 Go
BRIN (x2) vs seqscan
BRIN (1.4x) vs Btree
BRIN (x2.4) vs seqscan
BRIN (1.4x) vs Btree

Pour comparer le volume de données lues et la durée d’exécution nous pouvons désactiver les index dans une transaction :

BEGIN;
DROP index ...;
explain (analyse,verbose,buffers) SELECT ...
rollback;

Pour le 1er test, le moteur choisit l’index-btree. En supprimant l’index b-tree il choisit l’index BRIN.

Pour les tests 2 et 3, le moteur choisit l’index BRIN, en supprimant l’index BRIN il choisit l’index b-tree.

Pour le dernier test j’ai rajouté d’autres mesures. En effet, en supprimant l’index BRIN le moteur va effectuer un seqscan (parcours de toute la table). Pour obtenir les mêmes comparaisons que les résultats précédents j’ai donc supprimé l’index BRIN et désactivé les parcours séquentiels (set enable_seqscan to ‘off’;)

Globalement on peut constater un gain de 30-40% dans les cas où beaucoup de résultats sont demandés. Le moteur lit moins de blocs lorsqu’il utilise les index BRIN, l’index b-tree étant volumineux, ses lectures sont coûteuses.

En revanche l’index b-tree s’avère particulièrement performant lorsque la requête est très sélective et que peu de résultats sont retournés. En effet, en utilisant un index BRIN, le moteur commence par lire l’intégralité de l’index. Puis il va lire un ensemble de blocs qui contiennent la valeur recherchée, certains ne contenants aucun résultat. Ces lectures supplémentaires se ressentent sur la durée d’exécution de la requête.

Performances en insertion

Vu que les index BRIN sont plus petits et leur durée de création plus courte, on peut se demander ce qu’il advient du surcoût de cet index lors d’insertion de données. Pour cela on va créer une table et mesurer l’insertion de 10 millions de lignes en fonction des index déjà présents sur la table. Afin de cibler le surcoût dû à la mise à jour des index, la table est non-journalisée, ceci permet d’éviter les écritures dans les journaux de transaction. L’autovacuum est également désactivé.

CREATE UNLOGGED TABLE brin_demo_2 (c1 INT);
INSERT INTO brin_demo_2 SELECT * FROM generate_series(1,10000000);
TRUNCATE brin_demo_2;

CREATE INDEX brin_demo_2_brin_idx ON brin_demo_2 USING brin (c1);
INSERT INTO brin_demo_2 SELECT * FROM generate_series(1,10000000);
DROP INDEX brin_demo_2_brin_idx;
TRUNCATE brin_demo_2;
 
CREATE INDEX brin_demo_2_brin_idx ON brin_demo_2 USING brin (c1) WITH (pages_per_range = 256);
INSERT INTO brin_demo_2 SELECT * FROM generate_series(1,10000000);
DROP INDEX brin_demo_2_brin_idx;
TRUNCATE brin_demo_2;
...

Voici les résultats obtenus :

insertionComme pour les chiffres sur les performances en lecture, ces chiffres ne représentent que les durées d’insertion sur mon matériel. La table, les index et les journaux de transaction sont sur le même disque physique, ce qui ralenti les opérations.

Cependant on peut constater qu’il est moins coûteux d’insérer des données dans une table avec un index BRIN qu’avec un index b-tree. On constate également qu’il n’y a pas d’écart significatif entre les différents types d’index BRIN.

Conclusion

Cette série d’articles a permis de présenter les principes des index BRIN. puis leur fonctionnement à travers des exemples simples.

Ensuite nous avons vu l’importance de la corrélation pour exploiter pleinement ces index. Enfin, nous avons essayé de mesurer le gain que pouvait apporter cet index sur de multiples aspect (maintenance, performance en lecture et insertion).

Décrire le fonctionnement d’un index en simplifiant sa représentation est un exercice compliqué. On peut vite sacrifier le fond à la forme. Présenter des chiffres est également délicat tellement ils peuvent dépendre du contexte. J’ai fait l’effort de détailler comment je les ai obtenu afin que chacun puisse reproduire ses propres tests. L’idée est de donner un aperçu des cas d’utilisation de ce type d’index.

Globalement il faut retenir que les index BRIN sont utiles pour les tables volumineuses et où la corrélation avec l’emplacement des données est importante. Ils seront plus lents que les index b-tree lorsque la recherche nécessite de parcourir peu de blocs. Ils seront un peu plus rapide que les index b-tree dans les situations où le moteur doit lire beaucoup de blocs (moins de blocs à lire dans l’index).

L’étude de cet index ouvre d’autres pistes de réflexion. Comme la prise en compte de la corrélation dans le calcul du coût. J’avais également pensé à la possibilité d’utiliser un index pour créer un autre index.

Dans l’exemple avec la table volumineuse (150Go). Si on souhaite créer un index partiel sur le mois précédent, le moteur va parcourir l’intégralité de la table pour créer d’index. On pourrait envisager créer l’index b-tree en utilisant l’index BRIN pour ne parcourir que les lignes correspondant au moins précédent.

Partager

par Adrien Nayrat le jeudi 21 avril 2016 à 20h58

dimanche 13 mars 2016

Guillaume Lelarge

Fin de la traduction du manuel de la 9.5

Beaucoup de retard pour cette fois, mais malgré tout, on a fini la traduction du manuel 9.5 de PostgreSQL. Évidemment, tous les manuels ont aussi été mis à jour avec les dernières versions mineures.

N'hésitez pas à me remonter tout problème sur la traduction.

De même, j'ai pratiquement terminé la traduction des applications. Elle devrait être disponible pour la version 9.5.2 (pas de date encore connue).

par Guillaume Lelarge le dimanche 13 mars 2016 à 10h41

mardi 8 mars 2016

Sébastien Lardière

Dates à retenir

Trois dates à retenir autour de PostgreSQL : 17 mars, à Nantes, un premier meetup, dans lequel j'évoquerai les nouveautés de PostgreSQL 9.5. 31 mars, à Paris, où j'essayerai de remonter le fil du temps de bases de données. 31 mai, à Lille, où je plongerai dans les structures du stockage de PostgreSQL. Ces trois dates sont l'occasion... Lire Dates à retenir

par Sébastien Lardière le mardi 8 mars 2016 à 08h30

samedi 6 février 2016

Guillaume Lelarge

Début de la traduction du manuel 9.5

J'ai enfin fini le merge du manuel de la version 9.5. Très peu de temps avant la 9.5, le peu de temps que j'avais étant consacré à mon livre. Mais là, c'est bon, on peut bosser. D'ailleurs, Flavie a déjà commencé et a traduit un paquet de nouveaux fichiers. Mais il reste du boulot. Pour les intéressés, c'est par là : https://github.com/gleu/pgdocs_fr/wiki/Translation-9.5

N'hésitez pas à m'envoyer toute question si vous êtes intéressé pour participer.

par Guillaume Lelarge le samedi 6 février 2016 à 12h18

jeudi 4 février 2016

Rodolphe Quiédeville

Indexer pour rechercher des chaines courtes dans PostgreSQL

Les champs de recherche dans les applications web permettent de se voir propooser des résultats à chaque caractère saisies dans le formulaire, pour éviter de trop solliciter les systèmes de stockage de données, les modules standards permettent de définir une limite basse, la recherche n'étant effective qu'à partir du troisième caractères entrés. Cette limite de 3 caractères s'explique par la possibilité de simplement définir des index trigram dans les bases de données, pour PostgreSQL cela se fait avec l'extension standard pg_trgm, (pour une étude détaillé des trigrams je conseille la lecture de cet article).

Si cette technique a apporté beaucoup de confort dans l'utilisation des formulaires de recherche elle pose néanmoins le problème lorsque que l'on doit rechercher une chaîne de deux caractères, innoportun, contre-productif me direz-vous (je partage assez cet avis) mais imaginons le cas de madame ou monsieur Ba qui sont présent dans la base de données et dont on a oublié de saisir le prénom ou qui n'ont pas de prénom, ils ne pourront jamais remonter dans ces formulaires de recherche, c'est assez fâcheux pour eux.

Nous allons voir dans cet article comment résoudre ce problème, commençons par créer une table avec 50000 lignes de données text aléatoire :

CREATE TABLE blog AS SELECT s, md5(random()::text) as d 
   FROM generate_series(1,50000) s;
~# SELECT * from blog LIMIT 4;
 s |                 d                
---+----------------------------------
 1 | 8fa4044e22df3bb0672b4fe540dec997
 2 | 5be79f21e03e025f00dea9129dc96afa
 3 | 6b1ffca1425326bef7782865ad4a5c5e
 4 | 2bb3d7093dc0fffd5cebacd07581eef0
(4 rows)

Admettons que l'on soit un fan de musique des années 80 et que l'on recherche si il existe dans notre table du texte contenant la chaîne fff.

~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%fff%';

                                             QUERY PLAN                                              
-----------------------------------------------------------------------------------------------------
 Seq Scan on blog  (cost=0.00..1042.00 rows=5 width=37) (actual time=0.473..24.130 rows=328 loops=1)
   Filter: (d ~~ '%fff%'::text)
   Rows Removed by Filter: 49672
 Planning time: 0.197 ms
 Execution time: 24.251 ms
(5 rows)

Sans index on s'en doute cela se traduit pas une lecture séquentielle de la table, ajoutons un index. Pour indexer cette colonne avec un index GIN nous allons utiliser l'opérateur gin_trgm_ops disponible dans l'extension pg_trgm.

~# CREATE EXTENSION pg_trgm;
CREATE EXTENSION
~# CREATE INDEX blog_trgm_idx ON blog USING GIN(d gin_trgm_ops);
CREATE INDEX
~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%fff%';
                                                       QUERY PLAN                                                        
-------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on blog  (cost=16.04..34.46 rows=5 width=37) (actual time=0.321..1.336 rows=328 loops=1)
   Recheck Cond: (d ~~ '%fff%'::text)
   Heap Blocks: exact=222
   ->  Bitmap Index Scan on blog_trgm_idx  (cost=0.00..16.04 rows=5 width=0) (actual time=0.176..0.176 rows=328 loops=1)
         Index Cond: (d ~~ '%fff%'::text)
 Planning time: 0.218 ms
 Execution time: 1.451 ms

Cette fois l'index a pu être utilisé, on note au passage que le temps de requête est réduit d'un facteur 20, mais si l'on souhaite désormais rechercher une chaîne de seulement 2 caractères de nouveau une lecture séquentielle a lieu, notre index trigram devient inefficace pour cette nouvelle recherche.

~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%ff%';
                                               QUERY PLAN                                                
---------------------------------------------------------------------------------------------------------
 Seq Scan on blog  (cost=0.00..1042.00 rows=3030 width=37) (actual time=0.016..11.712 rows=5401 loops=1)
   Filter: (d ~~ '%ff%'::text)
   Rows Removed by Filter: 44599
 Planning time: 0.165 ms
 Execution time: 11.968 ms

C'est ici que vont intervenir les index bigram, qui comme leur nom l'index travaille sur des couples et non plus des triplets. En premier nous allons tester pgroonga, packagé pour Debian, Ubuntu, CentOS et d'autres systèmes exotiques vous trouverez toutes les explications pour le mettre en place sur la page d'install du projet.

Les versions packagées de la version 1.0.0 ne supportent actuellement que les versions 9.3 et 9.4, mais les sources viennent d'être taguées 1.0.1 avec le support de la 9.5.

CREATE EXTENSION pgroonga;

La création de l'index se fait ensuite en utilisant

~# CREATE INDEX blog_pgroonga_idx ON blog USING pgroonga(d);
CREATE INDEX
~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%ff%';
                                                           QUERY PLAN                                                            
---------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on blog  (cost=27.63..482.51 rows=3030 width=37) (actual time=3.721..5.874 rows=2378 loops=1)
   Recheck Cond: (d ~~ '%ff%'::text)
   Heap Blocks: exact=416
   ->  Bitmap Index Scan on blog_pgroonga_idx  (cost=0.00..26.88 rows=3030 width=0) (actual time=3.604..3.604 rows=2378 loops=1)
         Index Cond: (d ~~ '%ff%'::text)
 Planning time: 0.280 ms
 Execution time: 6.230 ms

On retrouve une utilisation de l'index, avec comme attendu un gain de performance.

Autre solution : pg_bigm qui est dédié plus précisément aux index bigram, l'installation se fait soit à partie de paquets RPM, soit directement depuis les sources avec une explication sur le site, claire et détaillée. pg_bigm supporte toutes les versions depuis la 9.1 jusqu'à 9.5.

~# CREATE INDEX blog_bigm_idx ON blog USING GIN(d gin_bigm_ops);
CREATE INDEX
~# EXPLAIN ANALYZE SELECT * FROM blog WHERE d like '%ff%';
                                                         QUERY PLAN                                                          
-----------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on blog  (cost=35.48..490.36 rows=3030 width=37) (actual time=2.121..5.347 rows=5401 loops=1)
   Recheck Cond: (d ~~ '%ff%'::text)
   Heap Blocks: exact=417
   ->  Bitmap Index Scan on blog_bigm_idx  (cost=0.00..34.73 rows=3030 width=0) (actual time=1.975..1.975 rows=5401 loops=1)
         Index Cond: (d ~~ '%ff%'::text)
 Planning time: 4.406 ms
 Execution time: 6.052 ms

Sur une table de 500k tuples la création de l'index prend 6,5 secondes pour bigm contre 4,8 pour pgroonga ; en terme de lecture je n'ai pas trouvé de pattern avec de réelle différence, bien pgroonga s'annonce plus rapide que pg_bigm, ce premier étant plus récent que le second on peut s'attendre à ce qu'il ait profité de l'expérience du second.

Coté liberté les deux projets sont publiés sous licence PostgreSQL license.

La réelle différence entre les deux projets est que Pgroonga est une sous partie du projet global Groonga qui est dédié à la recherche fulltext, il existe par exemple Mgroonga dont vous devinerez aisément la cible, pg_bigm est lui un projet autonome qui n'implémente que les bigram dans PostgreSQL.

Vous avez désormais deux méthodes pour indexer des 2-gram, prenez garde toutefois de ne pas en abuser.

La version 9.4.5 de PostgreSQL a été utilisée pour la rédaction de cet article.

par Rodolphe Quiédeville le jeudi 4 février 2016 à 08h38

lundi 1 février 2016

Guillaume Lelarge

Parution de mon livre : "PostgreSQL, architecture et notions avancées"

Après pratiquement deux ans de travail, mon livre est enfin paru. Pour être franc, c'est assez étonnant de l'avoir entre les mains : un vrai livre, avec une vraie reliure et une vraie couverture, écrit par soi. C'est à la fois beaucoup de fierté et pas mal de questionnements sur la façon dont il va être reçu.

Ceci étant dit, sans savoir si le livre sera un succès en soi, c'est déjà pour moi un succès personnel. Le challenge était de pouvoir écrire un livre de 300 pages sur PostgreSQL, le livre que j'aurais aimé avoir entre les mains quand j'ai commencé à utiliser ce SGBD il y a maintenant plus de 15 ans sous l'impulsion de mon ancien patron.

Le résultat est à la hauteur de mes espérances et les premiers retours sont très positifs. Ce livre apporte beaucoup d'explications sur le fonctionnement et le comportement de PostgreSQL qui, de ce fait, n'est plus cette espèce de boîte noire à exécuter des requêtes. La critique rédigée par Jean-Michel Armand dans le GNU/Linux Magazine France numéro 190 est vraiment très intéressante. Je suis d'accord avec son auteur sur le fait que le début est assez ardu : on plonge directement dans la technique, sans trop montrer comment c'est utilisé derrière, en production. Cette partie-là n'est abordée qu'après. C'est une question que je m'étais posée lors de la rédaction, mais cette question est l'éternel problème de l'oeuf et de la poule ... Il faut commencer par quelque chose : soit on explique la base technique (ce qui est un peu rude), puis on finit par montrer l'application de cette base, soit on fait l'inverse. Il n'y a certainement pas une solution meilleure que l'autre. Le choix que j'avais fait me semble toujours le bon, même maintenant. Mais en effet, on peut avoir deux façons de lire le livre : en commençant par le début ou en allant directement dans les chapitres thématiques.

Je suis déjà prêt à reprendre le travail pour proposer une deuxième édition encore meilleure. Cette nouvelle édition pourrait se baser sur la prochaine version majeure de PostgreSQL, actuellement numérotée 9.6, qui comprend déjà des nouveautés très excitantes. Mais cette édition ne sera réellement intéressante qu'avec la prise en compte du retour des lecteurs de la première édition, pour corriger et améliorer ce qui doit l'être. N'hésitez donc pas à m'envoyer tout commentaire sur le livre, ce sera très apprécié.

par Guillaume Lelarge le lundi 1 février 2016 à 17h57

mercredi 13 janvier 2016

Sébastien Lardière

Version 9.5 de PostgreSQL - 3

Une nouvelle version majeure de PostgreSQL est disponible depuis le 7 janvier. Chacune des versions de PostgreSQL ajoute son lot de fonctionnalités, à la fois pour le développeur et l'administrateur. Cette version apporte de nombreuses fonctionnalités visant à améliorer les performances lors du requêtage de gros volumes de données.

Cette présentation en trois billets introduit trois types de fonctionnalités :

Ce dernier billet de la série liste quelques paramètres de configuration qui font leur apparition dans cette nouvelle version.Suivi de l'horodatage des COMMITLe paramètre track_commit_timestamp permet de marquer dans les journaux de transactions ("WAL") chaque validation ("COMMIT") avec la date et l'heure du serveur. Ce paramètre est un booléen, et... Lire Version 9.5 de PostgreSQL - 3

par Sébastien Lardière le mercredi 13 janvier 2016 à 15h12

Rodolphe Quiédeville

Index multi colonnes GIN, GIST

Ce billet intéressera tous les utilisateurs de colonnes de type hstore ou json avec PostgreSQL. Bien que celui-ci prenne pour exemple hstore il s'applique également aux colonnes json ou jsonb.

Commençons par créer une table et remplissons là avec 100 000 lignes de données aléatoires. Notre exemple représente des articles qui sont associés à un identifiant de langue (lang_id) et des tags catégorisés (tags), ici chaque article peut être associé à un pays qui sera la Turquie ou l'Islande.

~# CREATE TABLE article (id int4, lang_id int4, tags hstore);
CREATE TABLE
~# INSERT INTO article 
SELECT generate_series(1,10e4::int4), cast(random()*20 as int),
CASE WHEN random() > 0.5 
THEN 'country=>Turquie'::hstore 
WHEN random() > 0.8 THEN 'country=>Islande' ELSE NULL END AS x;
INSERT 0 100000

Pour une recherche efficace des articles dans une langue donnée nous ajountons un index de type B-tree sur la colonne lang_id et un index de type GIN sur la colonne tags.

~# CREATE INDEX ON article(lang_id);
CREATE INDEX
~# CREATE INDEX ON article USING GIN (tags);
CREATE INDEX

Nous avons maintenant nos données et nos index, nous pouvons commencer les recherches. Recherchons tous les articles écrit en français (on considère que l'id du français est le 17), qui sont associés à un pays (ils ont un tag country), et analysons le plan d'exécution.

~# EXPLAIN ANALYZE SELECT * FROM article WHERE lang_id=17 AND tags ? 'country';
                                                                QUERY PLAN                                                                
------------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on article  (cost=122.42..141.21 rows=5 width=35) (actual time=12.348..13.912 rows=3018 loops=1)
   Recheck Cond: ((tags ? 'country'::text) AND (lang_id = 17))
   Heap Blocks: exact=663
   ->  BitmapAnd  (cost=122.42..122.42 rows=5 width=0) (actual time=12.168..12.168 rows=0 loops=1)
         ->  Bitmap Index Scan on article_tags_idx  (cost=0.00..12.75 rows=100 width=0) (actual time=11.218..11.218 rows=60051 loops=1)
               Index Cond: (tags ? 'country'::text)
         ->  Bitmap Index Scan on article_lang_id_idx  (cost=0.00..109.42 rows=4950 width=0) (actual time=0.847..0.847 rows=5016 loops=1)
               Index Cond: (lang_id = 17)
 Planning time: 0.150 ms
 Execution time: 14.111 ms
(10 rows)

On a logiquement 2 parcours d'index, suivis d'une opération de combinaison pour obtenir le résultat final. Pour gagner un peu en performance on penserait naturellement à créer un index multi colonnes qui contienne lang_id et tags, mais si vous avez déjà essayé de le faire vous avez eu ce message d'erreur :

~# CREATE INDEX ON article USING GIN (lang_id, tags);
ERROR:  42704: data type integer has no default operator class for access method "gin"
HINT:  You must specify an operator class for the index or define a default operator class for the data type.
LOCATION:  GetIndexOpClass, indexcmds.c:1246

Le HINT donnne une piste intéressante, en effet les index de type GIN ne peuvent pas s'appliquer sur les colonnes de type int4 (et bien d'autres).

La solution réside dans l'utilisation d'une extension standard, qui combine les opérateurs GIN et B-tree, btree-gin, précisons tout de suite qu'il existe l'équivalent btree-gist.

Comme toute extension elle s'installe aussi simplement que :

~# CREATE EXTENSION btree_gin;
CREATE EXTENSION

Désormais nous allons pouvoir créer notre index multi-colonne et rejouer notre requête pour voir la différence.

~# CREATE INDEX ON article USING GIN (lang_id, tags);
CREATE INDEX
~# EXPLAIN ANALYZE SELECT * FROM article WHERE lang_id=17 AND tags ? 'country';
                                                             QUERY PLAN                                                              
-------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on article  (cost=24.05..42.84 rows=5 width=35) (actual time=1.983..3.777 rows=3018 loops=1)
   Recheck Cond: ((lang_id = 17) AND (tags ? 'country'::text))
   Heap Blocks: exact=663
   ->  Bitmap Index Scan on article_lang_id_tags_idx  (cost=0.00..24.05 rows=5 width=0) (actual time=1.875..1.875 rows=3018 loops=1)
         Index Cond: ((lang_id = 17) AND (tags ? 'country'::text))
 Planning time: 0.211 ms
 Execution time: 3.968 ms
(7 rows)

A la lecture de ce deuxième explain le gain est explicite, même avec un petit jeu de données le coût estimé est divisé par 3, l'on gagne une lecture d'index et une opération de composition. Maintenant nous pouvons supprimer les 2 autres index pour ne conserver que celui-ci.

par Rodolphe Quiédeville le mercredi 13 janvier 2016 à 14h11

mardi 12 janvier 2016

Sébastien Lardière

Version 9.5 de PostgreSQL - 2

Une nouvelle version majeure de PostgreSQL est disponible depuis le 7 janvier. Chacune des versions de PostgreSQL ajoute son lot de fonctionnalités, à la fois pour le développeur et l'administrateur. Cette version apporte de nombreuses fonctionnalités visant à améliorer les performances lors du requêtage de gros volumes de données.

Cette présentation en trois billets introduit trois types de fonctionnalités :

Ce deuxième billet évoque donc les nouvelles méthodes internes du moteur, c'est-à-dire de nouveaux outils dont PostgreSQL dispose pour traiter les données.Index BRINIl s'agit d'un nouveau type d'index, créé pour résoudre des problèmes d'accès à de très gros volumes de données ; le cas d'usage est une table de log, dans laquelle les données sont... Lire Version 9.5 de PostgreSQL - 2

par Sébastien Lardière le mardi 12 janvier 2016 à 10h45

vendredi 8 janvier 2016

Sébastien Lardière

Version 9.5 de PostgreSQL

Une nouvelle version majeure de PostgreSQL est disponible depuis le 7 janvier. Chacune des versions de PostgreSQL ajoute son lot de fonctionnalités, à la fois pour le développeur et l'administrateur. Cette version apporte de nombreuses fonctionnalités visant à améliorer les performances lors du requêtage de gros volumes de données.

Cette présentation en trois billets introduit trois types de fonctionnalités :

Ce premier billet revient donc sur les ajouts au langage SQL de la version 9.5 de PostgreSQL.Ces ajouts portent sur de nombreux champs de fonctionnalités, de la sécurité aux gestions de performances en passant par la gestion des transactions.UPSERTCe mot clé désigne en réalité la possibilité d'intercepter une erreur de clé primaire sur un ordre... Lire Version 9.5 de PostgreSQL

par Sébastien Lardière le vendredi 8 janvier 2016 à 10h49